Topic 1 – Investigating Searches

▪ Use the Search Job Inspector to examine how a search was processed and troubleshoot performance

• Tool allows you to examine:
  ` - Overall stats of the search (e.g., recores processed/returned, processing time)
  • How the search was processed
  • Where Splunk spent its time
• Use to troubleshoot search's performance and understand impact of knowledge objects on processing(e.g., event types, tags, lookups)
• Any existing(i.e., not expired) search job can be inspected

Provides details on cost to retrieve results, such as:
- command.search.index : Time to search the index for the loaction to read in rawdata files
- Tcommand.search.filter : Time to filter out events that do not match
- command.search.rawdata : Time to read events from the rawdata files

index=web sourcetype=access_combined

▪ Use SPL commenting to help identify and isolate problems

* SPL Commenting

• Use three backicks(```) before and after your commnet

index=security soucetype=linux_secure 
| ```single-series column chart```
chart count over vendor_action

• Comment out portions of your search to help identify and isolate problems

index=security sourcetype=linux_secure
```| single-series column chart
	chart count over vendor_action```

• To make very long SPL easier to read, add comments directly after the pipe(|)

index=security sourcetype=linux_secure "failed password" earliest=-14d@d latest=@d
| ```line chart with week-to-week comparison``` timechart span=1d count as Failures
| timewrap 1w
| rename _time as Day
| eval Day = strftime(Day, "%A")

Topic 2 – Splunk Architecture

▪ Understand the role of search heads, indexers, and forwarders in a Splunk deployment

▪ Understand how the components of a bucket (.tsidx and journal.gz files) are used

▪ Understand how bloom filters are used to improve search speed

Topic 3 – Streaming and Non-Streaming Commands

▪ Describe the parts of a search string

▪ Understand the use of centralized vs. distributable commands

* Transforming Commands

- Operate on an entire result set of data 
	| stats
    | timechart
    | chart
    |top
    |rare

* Centralized Streaming Commands

- Stateful Streaming Commands
	| eval
- if preceded by commands that run on a search head, all will run on a search head

▪ Create more efficient searches

index=security failed user=root
| timechart count span=1h
| stats avg(count) as HourlyAverage

index=network sourcetype=cisco_wsa_squid
| eval Risk = case(x_wbrs_score >= 3, "1 Safe", x_wbrs_score >= 0, "3 Neutral", x_wbrs_score >= -5, "4 Dangerous", 1==1, "Not Known")
| timechart count by Risk

index=network sourcetype=cisco_wsa_squid usage="Personal" OR usage="Violation"
| stats count as connections by suspect, usage
| rename username as suspect

index=network sourcetype=cisco_wsa_squid usage="Personal" OR usage="Violation"
| rename username as suspect
| stats count as connections by suspect, usage

Topic 4 – Breakers and Segmentation

▪ Understand how segmenters are used in Splunk

▪ Use lispy to reduce the number of events read from disk

Topic 5 – Commands and Functions for Troubleshooting

▪ Using the fieldsummary command

... | fieldsummary [maxvals=num] [field-list]

maxvals specifies the maximum number of unique values to display for each field (oprional; defaults to 100)

• fieldsummary Command Output Fields
  - field : field name
  • count : number of events with that field
  • distinct_count : number of unique values in field
  • is_exact : boolean (0 or 1) indicates whether distinct_count is exact
  • max (if field numeric)
  • mean (if field is numeric)
  • min (if field is numeric)
  • numeric_count : count of numeric values in field
  • stdev(if field numeric)
  • values : distinct values of field and count of each value

▪ Using the makeresults command

makeresults Command | makeresults
- Cenerally used with one or more eval commands
- Must ve first command in search succeeding a | pipe

| makeresluts
| eval raw = "*"
| rex field=raw "\[(?<foo>.*)\]"

▪ Using informational functions with the eval command

o the isnull function
o the typeof function