class: center, middle # Welcome to DevOps State College --- # Why DevOps State College? * Build a community and make connections * Exchange ideas * Learn * Promote DevOps culture * Fun * Not tied to one vendor/technology --- # What is DevOps? Grand unification of philosophy around how to manage Development (programmers, application analysts, application owners, project managers) and IT Operations (system admins, network admins, security, data center, storage, database admin) in a tightly-integrated way. DevOps is the belief that working together as a collaborative team will produce better results, and break down barriers and finger pointing. --- # Future Presentations --- # Job Opportunities * AppliedTrust * AppliedTrust is hiring in Philadelphia, Dallas and Boulder offices. * For more information, see https://www.appliedtrust.com/jobs/ or talk to me afterwards --- # A brief history of logging --- # stdout - Where it all begins * Send messages to user when things break * Could be triggered to only output errors if a flag was set * errors get mixed in with regular output making it easy to miss --- # stderr - Improving your debugging * Logs output to the screen still by default * Could be separated with I/O redirection to send output to a separate TTY or file * unless output is stored in a file, hard to go back and find errors --- # Writing to files * Lets an application write error messages to a dedicated log file sysadmins could review * Can have different verbosity levels set in app for how much you want in logs * File can be rotated using log rotate * On a server with a bunch of applications, where are all the log files? --- # Syslog - the perfect solution * Writes messages to a socket or port instead of opening a file * Syslog server can determine what kind of messages are important to save and which to discard * Logs can be configured to write remotely, centralizing logging * Can use regular expressions to find interesting events in logs * Multi-line logs get converted into separate lines * Hard 1K limit of log messages without recompiling --- # Logstash - parsing those log files * Can parse out multi-line logs using rules * can tag messages to make it easier to find intersting events --- # Elasticsearch - Indexing that mess * Logstash can write directly to elasticsearch * searchability of logs using a query language * Makes searching through logs similar to writing SQL queries --- # Graylog - bringing it all together * Can listen using syslog and write data to elasticsearch * Can query elasticsearch to find the logs your looking for * Can generate pretty graphs and widgets for a logging dashboard * Has an easy to use query language for finding what you need --- # GELF - cutting out the middle man * Sends logs in a json format * App can write it's own tags * Structure is created and preserved throughout the entire process --- # systemd journal - structure out of the box * Get all your services logs structured * Have interesting data like cgroups tagged in your log messages * By default, pulls out logs line-by-line from stdout for services * Can customize services to use systemd-journal libraries to write better logs --- # journal2gelf - preserving the structure * journalctl is great for parsing logs on the host * we can preserve all that structure in graylog by sending logs via journal2gelf service --- # graygelf - archiving the good stuff * keep your elasticsearch data small * write out important events to files for archival purposes (sudo attempts, unauthorized login attempts, critical errors) * simple nodejs scripting language * proxy all of it into graylog to index --- # Journal or GELF? * For custom apps * a number of libraries in different languages for both * if a journal library exists, it gives more flexibility * gelfj is buggy and doesn't preserve the severity level --- # Beyond Linux - nxlog * native gelf support * can send events from windows event log * has multiline plugin support similar logstash * great for parsing those raw text files into gelf messages --- class: center, middle # LIVE DEMO