A data analyst once received 2 GB of messy CSV logs from a retail store; half of them had missing prices, mixed date formats, and duplicated entries. The manager wanted insights by tomorrow morning.

Instead of panicking, the analyst opened up their Python notebook, imported Pandas and NumPy, and started cleaning. Within hours, they had a structured, usable dataset that revealed sales patterns the business had never seen before.

That’s the power of Python for data cleaning; it turns chaos into clarity.

💼 Follow me on LinkedIn for insights, stories, and reflections on my learning journey.
🐦 Follow me on Twitter for short, bite-sized thoughts and daily tech learnings.

Why Data Cleaning Matters

Before you can analyse or visualise, your data needs to make sense.
Imagine trying to compute “average sales” when half your dataset has missing or wrong numbers; the results would mislead you completely.

That’s where libraries like Pandas and NumPy shine.
They give you the tools to:

• Handle missing data.

• Fix inconsistent formats.

• Remove duplicates.

• Transform raw data into a structured form.

Let’s explore how they help you do this - step by step.

🐼 Pandas: The Data Cleaning Workhorse

If Python had a cleaning superhero, it would be Pandas.
It’s built on top of NumPy and makes working with tabular data (like Excel or CSV) incredibly simple.

Key Features

• DataFrame: A table-like structure with rows and columns

• Easy I/O: Read and write from CSV, Excel, SQL, JSON

• Built-in cleaning functions: dropna(), fillna(), replace(), and more

Example: Handling Missing Data

import pandas as pd

#Load data

df = pd.read_csv("sales.csv")

#Check missing values

print(df.isnull().sum())

#Fill missing values

df['price'].fillna(df['price'].mean(), inplace=True)

#Remove duplicates

df.drop_duplicates(inplace=True)

What’s happening here:

• We’re identifying missing values.

• Replacing them with the mean of the column.

• Removing duplicate rows.

A once-messy CSV now becomes a clean, reliable dataset ready for analysis.

Pro Tips for Pandas

✅ Always start with df.info() and df.describe(), they reveal data types and missing patterns.
✅ Use df.rename(columns={}) to keep column names consistent.
✅ Convert date columns using pd.to_datetime() for easy filtering and sorting.

🔢 NumPy: The Backbone of Data Computation

While Pandas is great for tables, NumPy excels at fast and efficient numerical operations, the foundation of almost every Python data science library.

Why It’s Great for Cleaning

• Works efficiently with large datasets.

• Helps normalise or standardise numeric data.

• Handles missing or invalid numbers gracefully.

Example: Dealing with Outliers

import numpy as np

data = np.array([10, 12, 11, 300, 13, 15, 14])

#Calculate mean and standard deviation

mean = np.mean(data) std = np.std(data)

#Filter out values beyond 2 std deviations

filtered_data = data[np.abs(data - mean) < 2 * std]

print(filtered_data)

This code removes extreme values (outliers) that could skew your results.

Pro Tips for NumPy

✅ Use np.isnan() to handle NaN values efficiently.
✅ Leverage vectorised operations instead of loops for faster performance.
✅ Combine with Pandas: clean using Pandas, then process numerically with NumPy.

Pandas + NumPy = Data Cleaning Powerhouse

In practice, you’ll often use them together.
Here’s a mini real-world example:

import pandas as pd

import numpy as np

df = pd.read_csv("data.csv")

#Replace missing numeric values with column median

df['age'] = df['age'].replace(np.nan, df['age'].median())

#Normalize numerical column

df['salary'] = (df['salary'] - np.mean(df['salary'])) / np.std(df['salary'])

#Drop duplicates and reset index

df = df.drop_duplicates().reset_index(drop=True)

In just a few lines, you’ve:
✔️ Handled missing values
✔️ Standardised salary data
✔️ Cleaned duplicates

This combination forms the backbone of any serious data-prep workflow in Python.

💡 Best Practices for Clean, Reliable Data

✅ Always visualize missing data - tools like seaborn.heatmap(df.isnull()) help.
✅ Document every cleaning step - reproducibility is key in data work.
✅ Validate results - check means, counts, and unique values after cleaning.
✅ Automate repetitive cleaning tasks - small scripts save hours in the long run.

📚 References

• Pandas Official Documentation

• NumPy User Guide

• Real Python: Cleaning Data With Pandas

🤝 Community Corner

Data cleaning can sometimes feel like detective work - uncovering clues, testing assumptions, and fixing hidden errors.

What’s the messiest dataset you’ve ever cleaned?
Or what trick helped you handle missing values efficiently?

💬 Share your story or favourite Pandas/NumPy trick in the comments, your experience might save someone else’s late-night debugging session!

❓ FAQ: Data Cleaning in Python

1. What’s the difference between Pandas and NumPy?
Pandas handles structured tabular data; NumPy handles numerical arrays and math operations.

2. Is Pandas built on top of NumPy?
Yes, Pandas uses NumPy under the hood for fast numerical processing.

3. Can I clean large datasets with Pandas?
Yes, but for very large data, consider using Dask or PySpark for scalability.

4. How do I handle missing values?
Use dropna(), fillna(), or replace them with averages, medians, or a constant.

5. What’s the best way to remove duplicates?
Use df.drop_duplicates() and verify with df.duplicated().sum().

6. How can I handle inconsistent data types?
Use astype() to convert columns into consistent formats (e.g., strings to integers).

7. Should I clean data before visualisation?
Absolutely! Clean data ensures your visualisations and insights are accurate.

8. Do I need both Pandas and NumPy?
Most of the time, yes — Pandas for structure, NumPy for speed and numeric accuracy.

🌐 Let’s Connect and Keep the Conversation Going

If you enjoyed this article and want to explore more about Python, AI, and DevOps, let’s connect!

💼 Follow me on LinkedIn for insights, stories, and reflections on my learning journey.
🐦 Follow me on Twitter for short, bite-sized thoughts and daily tech learnings.

Let’s learn, build, and grow - one Python script at a time. 💻✨

When I first discovered Azure Cognitive Services, I was blown away by how easy Microsoft made it to add AI features into any app, without needing a PhD in machine learning.

As someone coming from a Cloud & DevOps background, AI once felt like a black box. But Cognitive Services changed that perception. It made me realise that AI isn’t just for data scientists, it’s for every developer who wants to build smarter apps.

So, let’s explore five examples that bring Azure Cognitive Services to life.

What Are Azure Cognitive Services?

In simple terms, Azure Cognitive Services is a collection of pre-built AI models by Microsoft that help apps see, hear, speak, understand, and decide.

You can think of it as plug-and-play AI.
You send data → Azure’s trained model analyses it → you get a smart response.

It’s divided into five key areas:

• Vision 👁️ (Image and video understanding)

• Speech 🗣️ (Speech-to-text, text-to-speech, and translation)

• Language 📝 (Text analytics, sentiment, and translation)

• Decision 🧮 (Anomaly detection, personalisation, moderation)

• Search 🔍 (Contextual web, video, and image search)

🌟 You can connect with me on LinkedIn and Twitter to have more discussion!

Example 1: Face Recognition with the Vision API

Imagine you’re building a secure door-access app. Instead of typing passwords, users can simply look at the camera.

That’s possible using the Face API in Cognitive Services.

With a few lines of code, you can detect:

• Who’s in the image

• Their emotions (happy, angry, surprised)

• Even match them with stored profiles

Pro Tip: Always handle biometric data carefully, use encryption and follow compliance best practices like GDPR.

Best Use Cases:

• Attendance systems

• Smart photo albums

• Emotion-based customer analysis

📘 Learn more about Azure Face API

Example 2: Speech-to-Text for Hands-Free Apps

Have you ever dictated a message using your phone? That’s exactly what Speech-to-Text API can do, but for your own app.

In one of my side projects, I integrated this service into a Python-based voice assistant. It converted spoken words into text in real-time, and the accuracy surprised me.

Best Practices:

• Use the correct locale (e.g., en-IN, en-US) for better accuracy.

• Test with diverse voice samples to reduce bias.

Best Use Cases:

• Meeting transcription tools

• Voice-driven chatbots

• Accessibility apps

📘 Azure Speech Service documentation

Example 3: Real-Time Language Translation

The Translator API helps break language barriers by translating text or speech into 70+ languages.

Imagine a global e-commerce website where you can instantly translate product reviews or chat messages.

Developer Tip:
Use custom translation to fine-tune model accuracy for domain-specific terms (like medical or legal vocabulary).

Best Use Cases:

• Multilingual chat systems

• Global product reviews

• Educational apps for language learners

Example 4: Text Analytics for Sentiment & Insights

If you’re analysing tweets or customer reviews, the Text Analytics API can instantly tell you whether the feedback is positive, negative, or neutral.

In one of my DevOps automation dashboards, I used it to analyse incident tickets, identifying which were marked with frustration words (“urgent”, “broken”, “down”) to prioritise alerts.

Best Practices:

• Pre-clean text to remove URLs and symbols.

• Combine with Power BI for real-time visualisations.

Best Use Cases:

• Customer feedback analysis

• Social media sentiment tracking

• Ticket prioritization

Example 5: Content Moderation for Safe Communities

In community platforms, keeping harmful content away is vital. The Content Moderator API uses machine learning to detect profanity, adult content, and personally identifiable information (PII).

Think of it as your digital bouncer for text, images, or videos.

Pro Tip: Combine automated moderation with human review for best accuracy.

Best Use Cases:

• Online forums and chat apps

• Comment moderation tools

• Social media platforms

📘 Azure Content Moderator

Best Practices When Using Cognitive Services

• Start small, test one API at a time.

• Secure your keys and endpoints with Azure Key Vault.

• Monitor usage in Azure Portal using metrics and alerts.

• Leverage Azure AI Studio to visualise, test, and fine-tune your models.

• Always consider data privacy and ethical AI practices.

🤝 Community Corner: Let’s Talk!

What’s the coolest AI feature you’ve built (or dream of building) using Azure Cognitive Services?

Drop your ideas or questions in the comments - I’d love to hear how you’re using AI to make your apps smarter.
You can also connect with me on LinkedIn and Twitter to continue the discussion!

❓ FAQ: Azure Cognitive Services for Beginners

1. Is Azure Cognitive Services free to start with?
Yes! Microsoft offers a free tier with limited transactions for most APIs.

2. Do I need AI/ML experience?
Not at all. These are pre-trained models; no data science background required.

3. How do I authenticate the APIs?
Use the API key and endpoint provided in the Azure Portal.

4. Can I integrate these APIs with Python or Node.js?
Yes, SDKs are available for multiple languages, including Python, C#, Java, and JavaScript.

5. Are Cognitive Services available globally?
Yes, Azure has global regions, but always choose the one closest to your users for low latency.

6. How secure is the data I send?
All data is encrypted in transit and at rest. You can also choose to store or not store processed data.

7. Can I customise or retrain models?
Yes, some services (like Custom Vision or Translator) let you train models with your own data.

8. What’s the difference between Cognitive Services and Azure AI Studio?
Cognitive Services are APIs; AI Studio helps you visualise, test, and integrate those APIs easily.

Directory Navigation

• ls = list files and directories in current directory

• ls -a = list all files and directories including hidden files.

• ls -l = list files and directories in long format.

• pwd = shows the present working directory.

• cd [dir_path] = change location to specified directory.

• cd ~ = directory to $HOME.

• cd .. = move up one directory level.

Files

• mkdir [dir_name]\= create a new directory.

• rm [file_name]\= remove a file.

• rm -r [directory_name]\= remove a directory recursively.

• cp [source_file] [destination_file]\= copy the contents of one file to another file.

• mv [source_file] [destination_file]\= move or rename files or directories.

• touch [file_name]\= create a new file.

• cat [file_name]\= show the contents of a file.

• cat [source_file] >> [destination_file]\= append file contents to another file.

• nano [file_name]\= open or create a file using the nano text editor.

• vi [file_name], vim [file_name]\= open or create a file using the Vi/Vim text editor.

• head [file_name] = show the first ten lines of a file.

• tail [file_name]\= show the last ten lines of a file.

Users and Groups

• sudo useradd [user_name]\= Create a new user account.

• sudo userdel [user_name\= Delete a user account.

• sudo usermod -aG [group_name] [user_name]\= Modify user information (add a user to a group).

• sudo passwd [user_name\= Change the current user's or another user's password.

• sudo groupadd [group_name]\= Add a new group.

• sudo groupdel [group_name]\= Delete a group.

• sudo [command]\= Temporarily elevate user privileges to superuser or root.

• su - [user_name]\= Switch the user account or become a superuser.

File Permissions

• chmod 777 [file_name]\= Assign read, write, and execute file permission to everyone (rwxrwxrwx).

• chown [user_name] [file_name]\= Change the ownership of a file with chown command.

• chown [user_name]:[group_name] [file_name]\= Change the owner and group ownership of a file.

• chgrp [group_name] [file/directory]\= Change file or directory group.

Processes

• ps\= List active processes.

• top\= See all running processes

• htop\= Interactive and colorful process viewer.

• kill [process_id]\= Terminate a Linux process under a given ID.

• killall [label]\= Terminate all processes with a given label.

• nohup [command] &\= Run a Linux process in the background.

• bg\= List and resume stopped jobs in the background.

• fg\= Bring the most recently suspended job to the foreground.

• fg [job]\= Bring a particular job to the foreground.

System Management

• uptime\= Display how long the system has been running, including the load average.

• hostname\= View system hostname

• hostname -i\= Show the IP address of the system

• date\= See current date and time.

• cal\= Show current calendar (month and day).

• whoami\= See which user you are using.

Network

• ifconfig\= Display IP addresses of all network interfaces.

• ping [remote_host]\= Ping remote host.

• netstat\= Show network statistics.

• nslookup [domain_name]\= Receive information about an internet domain.

SSH Login

• ssh [user_name]@[host]\= Connect to a remote host as a user via SSH.

• ssh [host]\= Securely connect to a host via SSH default port 22.

• ssh-keygen\= Generate SSH key pairs

• scp [file_name] [user_name]@[host]:[rem ote_path]\= Securely copy files between local and remote systems via SSH.

Disk Usage

• df -h\= Check free and used space on mounted systems.

• mount\= Show currently mounted file systems.

• mount [device_path] [mount_point]\= Mount a device.

File Transfer

• scp [source_file] [user]@[remote_host]:[de stination_path]\= Copy a file to a server directory securely.

• wget [link]\= Download files from FTP or web servers.

• curl [link]\= Transfer data to or from a server.

File Compression

• tar czf [archive.tar.gz]\= Create a .gz compressed tar archive.

• tar cf [archive.tar] [file/ directory]\= Archive an existing file or directory

• gzip [file_name], gunzip [file_name.gz]\= Compress or decompress .gz files.

• unzip [archive.zip]\= Extract a zip archive.

Packages

1. (Debian/Ubuntu)

• sudo apt update\= Update package list.

• sudo apt upgrade\= Upgrade installed packages.

• sudo apt install [package_name]\= Install an APT package.

• sudo apt remove [package_name]\= Remove an APT package.

2. (RedHat/CentOS/Fedora)

• sudo yum update\= Update package list and upgrade them.

• sudo yum install [package_name]\= Install a package

• sudo yum remove [package_name]\= Remove a package.

Searching

• find [path] -name [search_pattern]\= Find files and directories.

• grep [search_pattern] [file_name]\= Search for a specific pattern in a file.

• grep -r [search_pattern] [directory_name]\= Recursively search for a pattern in a directory.

• grep -i [search_pattern] [file_name]\= Case insensitive search.

• locate [name]\= Locate all files and directories related to a particular name.

• awk '[search_pattern] {print $0}' [file_name]\= Print all lines matching a pattern in a file.

• sed 's/[old_text]/ [new_text]/' [file_name]\= Find and replace text in a specified file.

• find [dir_name] -name [search_pattern]\= Find files by name.

Service Management

• systemctl start [service_name]\= Start a service.

• systemctl stop [service_name]\= Stop a service.

• systemctl restart [service_name]\= Restart a service.

• systemctl status [service_name]\= Check service status.

• systemctl enable [service_name]\= Enable a service.

• systemctl disable [service_name]\= Disable a service.

Shell Scripting

1. Structure

    #!/bin/bash
    echo "Hey! This is my Day14 of 90DaysOfDevOps"
   

2. Variables

      #!/bin/bash
      NAME="Rakshita"
      echo "Hey, $NAME"
   

3. Conditional Statements

      if [ condition ]; then
        # Commands
      elif [ condition ]; then
        # Commands
      else
        # Commands
      fi
   

4. Loops

      for i in {1..5}; do
        echo "Iteration $i"
      done
   
      while [ condition ]; do
        # Commands
      done
   

Additional useful shell commands

• man [command]\= Display a built-in manual for a command.

• history\= Print the command history used in the terminal.

• echo [text]\= Display the text.

• who\= Show who is logged on.

GIT-GITHUB COMMANDS CHEAT SHEET

HAPPY LEARNING!😃

In Git, a branch is a new/separate version of a main repository. Git branch are pointer to a snapshot of your changes. The master branch is a default branch in Git./

When you want to add a new feature or fix a bug, no matter how big or small it is, you create a new branch to make your changes. If the changes in new branch are well suited and tested, the new branch will be merged to master branch and applied in production.

Git Revert and Reset

Git Revert command helps you undo an existing commit. It does not delete any data in the process, rather it creates a new commit with the included files reverted to their previous state. So, your version control history moves forward while the state of your file moves backward.

Git Reset is a powerful command that is used to undo local chnages to the state of a git repository. It is generally used at the time of merge conflicts to reset the conflicted files to their original state.

These forms correspond to command line arguments --soft, --mixed, --hard.

--soft: Aims to change the HEAD (where the last commit is in your local machine) reference to a specific commit. For instance, if we realize that we forgot to add a file to the commit, we can move back using the --soft

--mixed: Any changes that have been undone from the Staging Index are moved to the Working Directory.

--hard: All the pending work in Staging area and Working Directory willbe lost.

Git Rebase and Merge

Git Rebase: Rebasing is the process of moving or combining a sequence of commits to a new base commit. Rebasing is most useful and easily visualised in the context of a feature branching workflow. The primary reason for rebasing is to maintain a linear project history.

Git Merge: It is used to combine the changes from one or more branches into the current branch. Git merge will associate a series of commits into one unified history. Generally, git merge is used to combine two branches.

Tasks

Task 1: Feature Development with Branches

1. Create a Branch and Add a Feature:

   • Add a text file called version01.txt inside the Devops/Git/ directory with “This is the first feature of our application” written inside.

       mkdir -p Devops/Git && cd Devops/Git && touch version01.txt
     

       vim version01.txt
     

     

   • Create a new branch from master.

       git checkout -b dev
     

     

     1. Commit your changes with a message reflecting the added feature.

         git add Devops/Git/version01.txt
         git commit -m "Added new feature"
        

        

     2. Push Changes to GitHub:

        • Push your local commits to the repository on GitHub.

            git push origin dev
          

          

          

     3. Add More Features with Separate Commits:

        Update version01.txt with the following lines, committing after each change:

        • 1st line: This is the bug fix in development branch

            echo "This is the bug fix in development branch" >> version01.txt
            git commit -am "Added feature2 in development branch"
          

          

        • 2nd line: This is gadbad code

            echo "This is gadbad code" >> version01.txt
            git commit -am "Added feature3 in development branch"
          

          

        • 3rd line: This feature will gadbad everything from now

            echo "This feature will gadbad everything from now" >> version01.txt
            git commit -am "Added feature4 in development branch"
          

          

4. Restore the File to a Previous Version:

   • Revert or reset the file to where the content should be “This is the bug fix in development branch”.

       git revert HEAD~2
     

Task 2: Working with Branches

1. Demonstrate Branches:

   • Create 2 or more branches and take screenshots to show the branch structure.

     

2. Merge Changes into Master:

   • Make some changes to the dev branch and merge it into master.

       git checkout master
       git merge dev
     

     

3. Practice Rebase:

   • Try rebasing and observe the differences.

       git rebase master
     

     

HAPPY LEARNING🌟

5. What is the difference between a local & remote repository? How to connect local to remote?

   Local repository: A local repository is a copy of the entire project’s history and codebase that resides on a developer’s machine. It allows developers to work offline, experiment with different ideas, and maintain a private workflow.

   Remote Repository: A remote repository is a central shared repository hosted on a server, typically accessible over the internet. Examples are GitHub, GitLab, or Bitbucket.

   Connect your local repository to remote repository

   • Make a repository on GitHub (remote) as we made above

   • Initialize an empty git repository:

    git init

• Add remote repository URL:

     git remote add origin <remote_repo_URL>

• Push to remote:

    git push origin main

Tasks

1. Task 1:

   • Set your user name and email address, which will be associated with your commits.

       # make directory
       mkdir DevOps && cd DevOps
     
       # initialize git repo
       git init
     
       # set username
       git config --global user.name "rakshita"
     
       #set email
       git config --global user.email "rakshitabelwal@gmail.com"
     

Task 2:

• Create a repository named "DevOps" on GitHub.

  

  

• Connect your local repository to the repository on GitHub.

  Connecting local repository to remote repository using Personal Access Token (PAT):

  • Create a PAT from your GitHub account-

Login to github -> Go to Settings -> Developer Settings -> Personal Access Token -> Token (Classic) -> Generate New Token -> Copy the token and save it somewhere

        git remote add origin <remote_repo_URL>
        git remote -v

        git remote set-url origin https://<PERSONAL_ACCESS_TOKEN>@github.com/RakshitaBelwal/DevOps.git
        git remote -v

• Create a new file in Devops/Git/Day-02.txt & add some content to it.

    mkdir -p DevOps/Git && cd DevOps/Git
    touch Day-02.txt
  

  

• Push your local commits to the repository on GitHub.

    git add Day-02.txt
    git commit -m "My first commit"
    git push origin master
  

  

Conclusion

Mastering the basics of Git and GitHub is essential for anyone involved in software development or version control. With these tools, developers can streamline their workflows, ensure code integrity, and collaborate more effectively, making Git and GitHub foundational skills in modern development practices.

HAPPY LEARNING🚀

Task 1: Checking Exit Status

• Write a script that attempts to create a directory and checks if the command was successful. If not, print an error message.

  1. Create a file:

      vim create_dir.sh
     

  2. Enter the script in file:

      #!/bin/bash
     
      # Create directory
      mkdir /home/ubuntu/day11
     
      # Check exit status
      if [ $? -ne 0 ]; then
              echo "Directory was not created"
              exit 1
      fi
              echo "Directory created successfully"
     

  3. Make it executable and execute. The OUTPUT will be:

     

Task 2: Using if Statements for Error Checking

Modify the script from Task 1 to include more commands (e.g., creating a file inside the directory) and use if statements to handle errors at each step.

#!/bin/bash

# Create directory inside directory
mkdir /home/ubuntu/day11/task2_dir
if [ $? -ne 0 ]; then
        echo "Failed to create Directory /home/ubuntu/day11/task2_dir "
        exit 1
fi

# Create file inside directory
touch /home/ubuntu/day11/task2_dir/newfile.txt
if [ $? -ne 0 ]; then
        echo "Failed to create file /home/ubuntu/day11/task2_dir/newfile.txt"
        exit 1
fi
        echo "Directory and file created successfully /home/ubuntu/day11/task2_dir/newfile.txt"

OUTPUT:

Task 3: Using trap for Cleanup

• Write a script that creates a temporary file and sets a trap to delete the file if the script exits unexpectedly.

    #!/bin/bash
  
    # Create a temporary file
    tempfile="temp_file.txt"
    touch $tempfile
  
    # Trap to delete temporary file
    trap "rm -f $tempfile; echo 'Temporary file deleted'" EXIT
  
    # Stimualte with sleep (10 seconds)
    echo "Script running..."
    sleep 10
  
    echo "Script completed successfully"
  

  OUTPUT: The script ran for 10 seconds and then temporary file deleted.

  

Task 4: Redirecting Errors

• Write a script that tries to read a non-existent file and redirects the error message to a file called error.log.

    #!/bin/bash
  
    # Read non-existing file and redirect error
    cat non_existing_file 2> error.log
  

  OUTPUT:

  

Task 5: Creating Custom Error Messages

• Modify one of the previous scripts to include custom error messages that provide more context about what went wrong.

    #!/bin/bash
  
    # Create directory
    mkdir /home/ubuntu/task5_dir
  
    if [ $? -eq 0 ]; then
            echo "Directory created successfully"
    else
            echo "Failed: Directory was not created for /home/ubuntu/task5_dir. Check if you have required permissions!"
    fi
  
    # Create a file inside directory
    touch /home/ubuntu/task5_dir/myfile5.txt
  
    if [ $? -eq 0 ]; then
            echo "File created successfully"
    else
            echo "Failed: File was not created for /home/ubuntu/task5_dir/myfile5.txt. Check, your directory already exist and the file has its required permissions! "
    fi
  

  OUTPUT: When the task successfully completed.

  

  OUTPUT: When the task failed to complete.

  

HAPPY SCRIPTING🌟

Scenario

You are a system administrator responsible for managing a network of servers. Every day, a log file is generated on each server containing important system events and error messages. As part of your daily tasks, you need to analyze these log files, identify specific events, and generate a summary report.

Task

Write a Bash script that automates the process of analyzing log files and generating a daily summary report. The script should perform the following steps:

1. Input: The script should take the path to the log file as a command-line argument.

2. Error Count: Analyze the log file and count the number of error messages. An error message can be identified by a specific keyword (e.g., "ERROR" or "Failed"). Print the total error count.

3. Critical Events: Search for lines containing the keyword "CRITICAL" and print those lines along with the line number.

4. Top Error Messages: Identify the top 5 most common error messages and display them along with their occurrence count.

5. Summary Report: Generate a summary report in a separate text file. The report should include:

   • Date of analysis

   • Log file name

   • Total lines processed

   • Total error count

   • Top 5 error messages with their occurrence count

   • List of critical events with line numbers

6. Optional Enhancement: Add a feature to automatically archive or move processed log files to a designated directory after analysis.

Tips

• Use grep, awk, and other command-line tools to process the log file.

• Utilize arrays or associative arrays to keep track of error messages and their counts.

• Use appropriate error handling to handle cases where the log file doesn't exist or other issues arise.

Solution:

1. Create a folder and then a log file inside:

   

2. In log_analyzer.sh insert the shell script:

    #!/bin/bash
   
    if [ $# -ne 1 ]; then
            echo "Usage: $0 /home/ubuntu/logs/log_file.log"
            exit 1
    fi
   
    log_file=$1
    summary_report="summary_$(date +%Y-%m-%d).txt"
   
    # check if the log file exists
    if [ ! -f "$log_file" ]; then
      echo "Error: Log file not found!"
      exit 1
    fi
   
    # Count the total number of lines in the log file
    total_lines=$(wc -l < "$log_file")
   
    # Count the number of error messages
    error_count=$(grep -c "ERROR" "$log_file")
   
    # Search for critical events
    critical_events=$(grep -n "CRITICAL" "$log_file")
   
    # Identify the top 5 most common error messages
    declare -A error_messages
    while IFS= read -r line; do
         if [[ "$line" == "ERROR" ]]; then
               ((error_messages["$message"]++))
         fi
    done < "$log_file"
   
    TOP_ERRORS=$(for msg in "${!ERROR_MESSAGES[@]}"; do
      echo "${ERROR_MESSAGES[$msg]}: $msg"
    done | sort -nr | head -5)
   
    # Generate summary report
    {
   
    echo "Date of analysis: $(date)"
    echo "Log file: $log_file"
    echo "Total line processed: $total_lines"
    echo "Total error count: $error_count"
    echo "Top 5 error messages:"
    echo "$TOP_ERRORS"
    echo "Critical events:"
    echo "$critical_events"
   
    } > "$summary_report"
   
    # Print the summary report to the console
    cat "$summary_report"
   
    # Optional: Archive or move the processed log file
    archive_dir="processed_logs"
    mkdir -p "$archive_dir"
    mv "$summary_report" "$archive_dir/"
   
    echo "Log file generated and saved to $summary_report"
   

3. Change the file permission to make it executable and execute:

   

4. Check summary report:

   

HAPPY SCRIPTING!🚀

Create a bash script that takes a directory path as a command-line argument and performs a backup of the directory. The script should create timestamped backup folders and copy all the files from the specified directory into the backup folder.

Additionally, the script should implement a rotation mechanism to keep only the last 3 backups. This means that if there are more than 3 backup folders, the oldest backup folders should be removed to ensure only the most recent backups are retained.

1. Create two directories- One is to write the shell script (scripts), another is to take the backup (backups):

    mkdir scripts backups
   

2. Enter into the scripts directory and make a file named backup_with_rotation.sh:

   

3. Insert the script:

    #!/bin/bash
   
    << readme
    This is a script to take a backup of directory
    usage:
    ./backup_with_rotation.sh <path of directory> <path of backup directory>
   
    readme
   
    source_dir=$1
    backup_dir=$2
    timestamp=$(date '+%Y-%m-%d-%H-%M-%S')
   
    function create_backup {
    zip -r "${backup_dir}/backup_${timestamp}.zip" "${source_dir}" > /dev/null
   
    if [ $? -eq 0 ]; then
   
            echo "Backup created successfully"
    else
            echo "Backup was not performed for $timestamp"
    fi
    }
   
    function perform_rotation {
                backups=($(ls -t "$backup_dir/backup_"*.zip))
   
                if [ "${#backups[@]}" -gt 3 ]; then
                        backups_to_remove=("${backups[@]:3}")
                        for backup in "${backups_to_remove[@]}";
                        do
                                rm "$backup"
                        done
                fi
    }
   
    create_backup
    perform_rotation
   

4. Change the permissions of this file to make it executable:

    chmod 700 backup_with_rotation.sh
   

5. Install zip:

       sudo apt install zip
   

6. Execute:

    ./backup_with_rotation.sh /home/ubuntu/scripts /home/ubuntu/backups
   

   Here,

   /home/ubuntu/scripts = path of directory

   /home/ubuntu/backups = path of backup directory

7. Take backup 4-5 times:

   

8. Check if the last 3 backups are there in your backups directory:

   

Conclusion

In this guide, we've walked through the creation of a robust bash script that automates the backup of directories with ease. By taking a directory path as a command-line argument, this script efficiently creates timestamped backup folders, ensuring that each backup is neatly organized and easily identifiable.

Moreover, we've implemented a rotation mechanism to keep only the last three backups, effectively managing storage space while maintaining the most recent data. This approach not only simplifies the backup process but also ensures that your system remains clutter-free.

Remember, automation is key to efficiency, and this backup script is a small yet powerful step in that direction.

Happy scripting!

In bash scripts, comments are used to add explanatory notes or disable certain lines of code. Your task is to create a bash script with comments explaining what the script does.

Make a file named task1.sh and insert the script:

  #!/bin/bash

  # This script prints about what comment does

  echo "The comment (#) are added with the purpose of making the source code easier for humans to understand, and are generally ignored by compilers and interpreters"

Make this file executable:

chmod 700 task1.sh

Execute the script:

./task1.sh

Task 2: Echo

The echo command is used to display messages on the terminal. Your task is to create a bash script that uses echo to print a message of your choice.

#!/bin/bash

echo "Hello, this is my Day 8 of 90DaysOfDevOPs challenge"

Run the script.

Task 3: Variables

Variables in bash are used to store data and can be referenced by their name. Your task is to create a bash script that declares variables and assigns values to them.

#!/bin/bash

name="Rakshita"
challenge="90DaysOfDevOps challenge"

echo "Hello, my name is $name and this is my day 8 of $challenge"

Run the script.

Task 4: Using Variables

Now that you have declared variables, let's use them to perform a simple task. Create a bash script that takes two variables (numbers) as input and prints their sum using those variables.

#!/bin/bash

num1=6
num2=9
sum=$((num1 + num2))

echo "The sum of $num1 and $num2 is $sum"

Run the script.

Task 5: Using Built-in Variables

Bash provides several built-in variables that hold useful information. Your task is to create a bash script that utilizes at least three different built-in variables to display relevant information.

#!/bin/bash

echo "Current user: $USER"

echo "Home directory: $HOME"

echo "Current working directory: $PWD"

Task 6: Wildcards

Wildcards are special characters used to perform pattern matching when working with files. Your task is to create a bash script that utilizes wildcards to list all the files with a specific extension in a directory.

Make few .txt files in current directory:

touch file1.txt file2.txt

Create another new file:

vim task6.sh

Enter the script:

#!/bin/bash

echo "Listing all .txt files in current directory"
ls *.txt

Make this file executable:

chmod 700 task6.sh

Run the script:

./task6.sh

Conclusion

In this series of tasks, we explored some fundamentals of bash scripting including comments, echo, commands,variables, build-in-variables and wildcards. This foundation euips us with the skills to write more complex and functional bash scripts, facilitating efficient and automated tasks in a Unix-like environment.

Happy learning!🚀

In simpler words, a package manager is a tool that allows users to install, remove, upgrade, configure and manage software packages on an operating system. The package manager can be a graphical application like a software center or a command line tool like apt-get or pacman.

What is Package?

A package contains all the necessary data required for the installation and maintenance of the software package. These packages are created by someone known as a package maintainer. A package maintainer takes care of the packages. They ensure active maintenance, bug fixes if any, and the final compilation of the package.

Different Kinds of Package Managers

There are different package managers for different Linux distributions. Here I have shown some of the popular package managers:

1. Advanced Packaging Tool (APT)

   The APT package manager is used in the Debian-based Linux distribution and its derivatives. APT package manager handles the .deb format packages. It install, remove, upgrade packages, and resolve dependencies automatically. It uses the apt, apt-get command line.

2. Yellow-Dog Updater Modified (YUM)

   YUM package manager is used in the RedHat-based Linux distribution and its derivatives. It is the default common package manager in RedHat and handles the .rpm formatted packages. Yum checks the dependencies with header files and package metadata and resolves dependencies while installing or updating packages.

3. Dandified YUM (DNF)

   DNF package manager is the successor and the top of the yum package manager. It is is also used in Red Hat-based distribution. Dnf also uses the .rpm file extension. Dnf provides faster performance, Parallel downloading, a rich command line interface, and enhanced dependency resolution by using a dependency solver.

4. Pacman

   Pacman is a utility that is used in Arch Linux distribution for managing software packages. Using simple compressed files, it maintains a text-based database. The file extension for the Pacman package manager is .pkg.tar.xz. Pacman is lightweight, fast, and high-speed packaging, it provides two types of repositories and automatically upgrades the package.

TASKS

How to install Docker and Jenkins using package managers on Ubuntu and CentOS.

DOCKER INSTALLATION ON UBUNTU

1. Update Packages:

    sudo apt-get update
   

2. Install Docker:

    sudo apt install docker.io -y
   

3. Verify the installation:

    docker --version
   

   ON CENTOS

4. Set up the repository:

    sudo yum install -y yum-utils
    sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo
   

5. Install Docker Engine, containerd, and Docker Compose:

    sudo yum install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
   

6. Verify the installation:

    docker --version
   

JENKINS INSTALLATION ON UBUNTU

1. Update package list:

    sudo apt-get update
   

2. Install Java (Jenkins requires Java):

    sudo apt-get install openjdk-11-jdk -y
   

3. Add the Jenkins repository key:

    curl -fsSL https://pkg.jenkins.io/debian/jenkins.io.key | sudo tee /usr/share/keyrings/jenkins-keyring.asc > /dev/null
   

4. Add the Jenkins repository:

    echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null
   

5. Update the package list again:

    sudo apt-get update
   

6. Install Jenkins:

    sudo apt-get install jenkins -y
   

7. Check the status:

    sudo systemctl status jenkins
   

ON CENTOS 9

1. Install java:

    sudo dnf install java-11-openjdk-devel -y
   

2. Install wget:

    sudo yum install wget
   

3. Install Jenkins:

    sudo wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
   
    sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
   
    sudo dnf upgrade
   
    sudo dnf install jenkins
   

4. Start Jenkins:

    sudo systemctl start jenkins
   

5. Check Status:

    sudo systemctl status jenkins
   

Systemctl and Systemd

systemd gives us the systemctl commands suite which is mostly used to enable services to start at boot time. We can also start, stop, reload, restart and check status of services with the help of systemctl .

We can do, for example, sudo systemctl enable service_name, and service_name will automatically start at boot time. We can also disable services not to start at boot time.

Tasks

1. Check Docker Service Status:

   • Check the status of the Docker service on your system (ensure you have completed the installation tasks above).

    sudo systemctl status docker

2. Manage Jenkins Service:

   • Stop the Jenkins service and post before and after screenshots.

Before stopping the jenkins service:

After stopping the jenkins service:

3. Systemctl vs. Service:

   • Differences between the systemctl and service commands.

Systemctl:- It works with systemd, the newer and more modern system and service manager. Provides detailed information about the service, including logs. Manages more than just services (e.g., system states like shutdown, reboot).

The syntax for systemctl is 'systemctl [command] [service_name]'. For example- To check the status of Docker: 'system status docker'

Service:- It works with older service managers like SysVinit and Upstart. Simpler and more traditional command structure. Simpler and more traditional command structure.

The syntax for service is 'service [service_name] [command]'. For example- To check the status of Docker: 'service docker status'

Additional Tasks

4. Automate Service Management:

   • Write a script to automate the starting and stopping of Docker and Jenkins services.

1. Make a file script.sh:

vim script.sh

2. Write a script:

#!/bin/bash

# Function to start services
start_services() {
    echo "Starting Docker and Jenkins services..."
    sudo systemctl start docker
    sudo systemctl start jenkins
    echo "Services started."
}

# Function to stop services
stop_services() {
    echo "Stopping Docker and Jenkins services..."
    sudo systemctl stop docker
    sudo systemctl stop jenkins
    echo "Services stopped."
}
# Check if the user provided an argument
if [ $# -eq 0 ]; then
    usage
fi

# Handle the input argument
case "$1" in
    start)
        start_services
        ;;
    stop)
        stop_services
        ;;
esac

3. Make the file executable:

chmod 700 script.sh

4. Run the script:

./script.sh start
OR
./script.sh stop

5. Enable and Disable Services:

   • Use systemctl to enable Docker to start on boot and disable Jenkins from starting on boot.

     Enable Docker to start on boot:

       sudo systemctl enable docker
     

     Disable Jenkins from starting on boot:

       sudo systemctl disable jenkins
     

6. Analyze Logs:

   • Use journalctl to analyze the logs of the Docker and Jenkins services.

1. Analyze Docker logs:

    sudo journalctl -u docker
   

2. Analyze Jenkins logs:

    sudo journalctl -u jenkins
   

Conclusion

Linux package managers and systemd (with systemctl) form the backbone of software and service management in modern Linux distributions. Package managers ensure that software installations are handled smoothly and consistently, managing dependencies and updates efficiently. Meanwhile, systemd and systemctl provide robust tools for controlling and managing system services, enhancing system reliability and maintainability. Understanding these tools is crucial for effective Linux system administration.

HAPPY LEARNING!

Access control list (ACL) provides an additional, more flexible permission mechanism for file systems. ACLs allow you to define different permissions for different users or groups beyond the standard owner/group/others model.

Using 'getfacl ' and 'setfacl '

setfacl and getfacl are used for setting up ACL and showing ACL respectively.

• 'getfacl '

The getfacl command is used to display the ACLs of a file or directory. This command shows the effective permissions of users and groups, as well as the default ACLs if they are set.

To display the ACLs of a file named rakshita.txt, we would use:

getfacl rakshita.txt

The output is:

# file: rakshita.txt
# owner: ubuntu
# group: ubuntu
user::rw-
group::rw-
other::r--

• # file: rakhsita.txt- The file we're looking at is named rakshita.txt.

• # ubuntu: ubuntu: The user who owns this file is ubuntu.

• # group: ubuntu: The group associated with this file is ubuntu.

• user: :rw-: The owner (ubuntu) has:

  • Read (r) permission: Can view the file.

  • Write (w) permission: Can modify the file.

  • No execute (-) permission: Cannot run the file as a program.

• group: :rw-: The group (ubuntu) has:

  • Read (r) permission: Can view the file.

  • Write (w) permission: Can modify the file.

  • No execute (-) permission: Cannot run the file as a program.

• other: :r--: All other users have:

  • Read (r) permission: Can view the file.

  • No write (-) permission: Cannot modify the file.

  • No execute (-) permission: Cannot run the file as a program.

• 'setfacl '

The setfacl command is used to set or modify the ACLs of a file or directory.

• To add permissions for user:

    setfacl -m "u:user:permissions" /path/to/file
  

• To add permissions for a group:

    setfacl -m "g:group:permissions" /path/to/file
  

• Remove a Specific ACL Entry:

      setfacl -x u:username /path/to/file
  

• To remove all entries:

setfacl -b path/to/file

Task: Create a directory and set specific ACL permissions for different users and groups. Verify the permissions using ''getfacl".

1. Create a directory:

    mkdir control
   

2. Grant read and write permissions to user1:

    setfacl -m u:user1:rw control
   

3. Grant read-only permissions to user2:

    setfacl -m u:user2:r control
   

4. Grant read and execute permissions to group1:

    setfacl -m g:group1:rx control
   

To verify the ACL permissions set on control, use the getfacl command:

getfacl control

Output:

# file: control
# owner: ubuntu
# group: ubuntu
user::rwx
user:user1:rw-
user:user2:r--
group::rwx
group:group1:r-x
mask::rwx
other::r-x

ADDITIONAL TASK:

Task: Create a script that changes the permissions of multiple files in a directory based on user input.

Create a file:

vim permissions.sh

Enter the bash secript int the file:

#!/bin/bash

echo "enter the directory path"
read directory_path

echo "enter permissions"
read permissions

chmod $permissions $directory_path
echo "permissions changed successfully"

Change the file permssions to make it executable:

chmod 700 permissions.sh

Run the file:

./permissions.sh

Understanding Sticky Bit, SUID, and SGID

1. Sticky Bit: A Sticky bit is a permission bit that is set on a file or a directory that lets only the owner of the file/directory or the root user to delete or rename the file.

2. SUID: SUID (Set User ID) is a special permission in Linux that allows a program to run with the permissions of the file's owner.

SGID: SGID (Set Group ID) is a special permission in Linux that affects both files and directories, but it works differently for each.

When SGID is set on an executable file, the program runs with the permissions of the file's group, rather than the user's group.

When SGID is set on a directory, any files created within that directory inherit the group ownership of the directory, rather than the group ownership of the user who created the file.

Task: Create examples demonstrating the use of sticky bit, SUID, and SGID.

1. To set the sticky bit permission on a directory, use the chmod command

   The numeric value for the sticky bit is 1.

   If you want to set the sticky bit permission on a directory named public with permission bits 777, run:

chmod 1777 public

2. The numeric value for SUID is 4.

   If you want to set the SUID permission on a file named script.sh that currently has the permission bits 755, you would run:

chmod 4755 script.sh

3. The numeric value for SGID is 2.

   If you want to set the SGID permission on a directory named shared with permission bits 755, you would execute:

chmod 2755 shared

1. Write a bash script create_directories.sh that, when executed with three arguments (directory name, start number of directories, and end number of directories), creates a specified number of directories with a dynamic directory name.

   • When executed as ./create_directories.sh day 1 90, it creates 90 directories as day1 day2 day3 ... day90.

    #!/bin/bash

    dir_name=$1
    start_day=$2
    end_day=$3

    mkdir $(eval echo $dir_name{$start_day..$end_day})

Make this file executable:

    chmod 700 create_directories.sh

Run the file:

    ./create_directories.sh day 1 90

Then list the directories:

    ls

Understanding Cron and Crontab : Automating Tasks on Linux

In the world of Linux, cron and crontab are essential tools for automating repetitive tasks. Whether you're a developer, a system administrator, or just a Linux enthusiast, knowing how to use these tools can save you time and effort.

What is Cron?

Cron is a time-based job scheduler in Unix-like operating systems. It runs in the background and executes commands at specified times and dates. Think of it as a personal assistant that handles your routine tasks for you.

What is Crontab?

Crontab (short for "cron table") is a file that contains the schedule of cron jobs. Each user can have their own crontab file, which lists the commands to be run and the times they should be executed.

Structure of a Crontab Entry

A typical crontab entry looks like this:

* * * * * command_to_be_executed
- - - - -
| | | | |
| | | | +----- Day of the week (0 - 7) (Sunday is both 0 and 7)
| | | +------- Month (1 - 12)
| | +--------- Day of the month (1 - 31)
| +----------- Hour (0 - 23)
+------------- Minute (0 - 59)

Each asterisk represents a field for time and date, and they can be replaced with specific values.

Understanding User Management

User management in Linux involves creating, modifying, and deleting user accounts. These accounts are essential for defining permissions, access levels, and individual user environments.

Creating and Managing User Accounts

1. Adding a New User

To add a new user, you can use the useradd command followed by the username. For example:

sudo useradd rakshita

2. Setting a Password

After creating a user, set a password using the passwd command:

sudo passwd rakshita

4. Deleting a User

To delete a user and their home directory, use the userdel command with the -r flag:

sudo userdel -r rakshita

Managing Groups

1. Adding a New Group

To add a new group, use the groupadd command:

sudo groupadd devops

2. Adding a User to a Group

To add a user to a group, use the usermod command with the -aG flags:

sudo usermod -aG devops rakshita

3. Viewing Group Membership

To view the groups a user belongs to, use the groups command:

groups rakshita

Understanding Permissions

Linux file permissions are represented by a set of three characters for the owner, group, and others. For example:

-rw-r--r--

The first character represents the file type ('-' for regular files, 'd' for directories). The next nine characters are divided into three sets of three:

1. Owner permissions (rw-): Read and write.

2. Group permissions (r--): Read-only.

3. Other permissions (r--): Read-only.

Permission Types

• r: Read permission. The file can be read.

• w: Write permission. The file can be modified.

• x: Execute permission. The file can be executed as a program.

Viewing File Permissions

To view the permissions of a file or directory, use the 'ls -l' command:

ls -l <filename>

This will display the permissions, owner, group, and other details about the file.

Changing File Permissions

The 'chmod' command is used to change the permissions of a file or directory.There are two ways to use chmod: symbolic mode and numeric mode.

Symbolic Mode

In symbolic mode, you use letters to specify the changes:

• u for user (owner)

• g for group

• o for others

• a for all

For example, to add execute permission for the owner:

chmod u+x <filename>

To remove write permission for the group:

chmod g-w <filename>

Numeric Mode

In numeric mode, you use a three-digit number to set the permissions. Each digit represents a different category of users (owner, group, others) and is the sum of the values for read (4), write (2), and execute (1) permissions.

For example:

• '7' (4+2+1) gives read, write, and execute permissions.

• '5' (4+1) gives read and execute permissions.

• '4' gives read permission only.

To set the permissions to rwxr-xr-- (755):

chmod 755 <filename>

Using 'chown' and 'chgrp'

The 'chown' command changes the owner of a file or directory:

sudo chown <newowner> <filename>

The 'chgrp' command changes the group of a file or directory:

sudo chgrp <newgroup> <filename>

We can also change both the owner and group at the same time:

sudo chown <newowner>:<newgroup> <filename>

Conclusion

Mastering user and file management in Linux is essential for maintaining a secure and efficient system. We explored how to create and manage users and groups using commands like useradd, usermod, and groupadd. We delved into file permissions, understanding how to view and modify them using chmod, both in symbolic and numeric modes.

By applying these concepts, you can ensure your Linux environment is well-organized and secure. Whether you're a beginner or an experienced user, these skills are fundamental to effective Linux system administration.

Happy managing, and keep exploring the powerful world of Linux! If you have any questions or tips, feel free to share them in the comments.

HAPPY LEARNING🚀

Shell scripting is a text file with a list of commands that instructs an operating system to perform certain tasks. A shell is an interface that interprets, processes, and executes these commands from the shell script. It can be particularly helpful to automate repetitive tasks, helping to save time and reduce human error. Each shell script is saved with .sh file extension e.g., myscript.sh.

2. What Shell Scripting means for DevOps?

Shell Scripting in DevOps means using scripts to automate repetitive tasks and make system management easier and more efficient. This helps DevOps engineers save time and ensure tasks are done consistently.

What Does Shell Scripting Do in DevOps?

1. Automates Tasks: Instead of doing tasks manually, you write a script to do them for you.

2. Saves Time: Once a script is written, you can run it anytime to perform the same task quickly.

3. Reduces Errors: Automating tasks minimizes the chances of making mistakes.

4. Ensures Consistency: Scripts perform tasks the same way every time, ensuring uniform results.

5. What is#!/bin/bash? Can we write#!/bin/shas well?

The line "#! /bin/bash" is called a shebang. It indicates the path to the interpreter that should be used to execute the script, in this case, "/bin/bash" for the Bash shell. It ensures that the script is interpreted by the specified shell, in this case, Bash. This line is placed at the beginning of a script to define the interpreter for proper execution.

Yes, we can write #!/bin/sh, but there are some differences to keep in mind:

1. Compatibility:

   • #!/bin/sh scripts are more portable because sh is available on all Unix-like systems.

   • Some advanced features of Bash may not be available or may behave differently in sh.

2. Features:

   • #!/bin/bash gives you access to all Bash-specific features, such as arrays, certain string operations, and more advanced scripting capabilities.

   • #!/bin/sh ensures that the script uses only the more basic features of the shell language, which can help make the script more portable across different systems.

3. Write a Shell Script that printsI will complete #90DaysOfDevOps challenge.

Open a text editor and create a file script.sh:

vim script.sh

Press i, add shebang and the script to be printed:

#!/bin/bash

echo "I will complete #90DaysOfDevOps challenge."

To save and exit, press 'esc', ':wq' and 'enter'.

Make the script executable for user/owner only:

chmod 700 script.sh

Run the script:

./script.sh

6. Write a Shell Script that takes user input, input from arguments, and prints the variables?

Create a arguments.sh file:

vim arguments.sh

Enter the script:

#!/bin/bash

# Take_user_input
echo "Enter your name:"
read user_input

# Input_from_argumemnts
arg1=$1
arg2=$2

# Print_variables
echo "User input: $user_input"
echo "First argument: $arg1"
echo "Second argument: $arg2"

Save and quit the file. Then change file permissions to make it executable by the user:

chmod 700 arguments.sh

Run the script:

./arguments.sh 1 2

Here;

• 1= argument 1 value.

• 2= argument 2 value.

7. Provide an example of an If-Else statement in Shell Scripting by comparing two numbers?

#!/bin/bash

# Define two numbers
num1=5
num2=10

# Compare the numbers
if [ $num1 -gt $num2 ]; then

    echo "Number 1 ($num1) is greater than Number 2 ($num2)"

elif [ $num1 -lt $num2 ]; then

    echo "Number 1 ($num1) is less than Number 2 ($num2)"
else

    echo "Number 1 ($num1) is equal to Number 2 ($num2)"
fi

Run the script.

Conclusion

We learned about basic Linux Shell Scripting and its fundamentals, which are essential to strengthen our foundation in Shell Scripting. We can write and practice more Shell commands and scripting on our own and by taking references from the internet.

Stay tuned for more in my #90daysdevops journey!

Happy Learning!✨

To display the content of file with line numbers in Linux, we use 'cat' command with the '-n' option.

cat -n filename.txt

2. Change the access permissions of files to make them readable, writable, and executable by the owner only?

To change the access permissions of file so that it is readable, writable and executable by the owner only, we use 'chmod' command with '700' mode.

chmod 700 filename.txt

Here's what 700 means:

• '7' for the owner: read (4), write (2), and execute (1) permissions (4+2+1 = 7).

• '0' for the group: no permissions.

• '0' for others: no permissions.

3. Check the last 10 commands you have run?

The 'history' command is generally used to display all the commands we used. By default, it shows all the commands we've run in the current session, but we can filter it to show only the last 10 commands:

history | tail -n 10

Here’s what each part of this command does:

• history: Displays the command history.

• |: Pipes the output of history to another command.

• tail -n 10: Shows the last 10 lines of the piped output.

4. Remove a directory and all its contents?

To remove a directory and all its contents, we use 'rm' command with the '-r' (recursive) option.

rm -r directory_name

5. Create afruits.txtfile, add content (one fruit per line), and display the content?

To create a file in Linux, we can use 'touch' command or directly open the file with a text editor like 'nano' or 'vim'.

Creating a file (fruits.txt):

Using 'touch'

touch fruits.txt

Adding content to 'fruits.txt':

Using text editor 'vim'

vim fruits.txt

After using above (vim) command we can add contents in 'fruits.txt' file. (Write different fruits name per line). To save and exit the file, press 'esc' then ':wq' and 'enter'.

Display the content of fruits.txt:

Using 'cat' command

cat fruits.txt

6. Add content indevops.txt(one in each line) - Apple, Mango, Banana, Cherry, Kiwi, Orange, Guava. Then, append "Pineapple" to the end of the file?

We can use text editor like 'vim' to create and enter fruits directly:

vim devops.txt

In the text editor, add the following lines:

Apple
Mango
Banana
Cherry
Kiwi
Orange
Guava

To save and exit the file, press 'esc' then ':wq' and 'enter'.

To append "Pineapple" to the file, use 'echo' command with the append operator '>>'.

echo "Pineapple" >> devops.txt

To verify use 'cat' command

cat devops.txt

7. Show the first three fruits from the file in reverse order?

To show first three fruits from file we use 'head' command

head -n 3 devops.txt

The 'head -n 3 devops.txt' command will output:

Apple 
Mango 
Banana

To reverse the order of the first three lines, we’ll use a combination of the 'head' and 'tac' commands. The 'tac' command is the reverse of the 'cat' command and displays lines in reverse order:

head -n 3 devops.txt | tac

This command first takes the top three lines of the file and then reverses their order. The output will be:

Banana
Mango
Apple

8. Show the bottom three fruits from the file, and then sort them alphabetically?

To display the bottom three lines of the devops.txt file, we can use the 'tail' command

tail -n 3 devops.txt

The output will be:

Orange
Guava
Pineapple

To sort these lines alphabetically, we can pipe the output of the 'tail' command into the 'sort' command:

tail -n 3 devops.txt | sort

The output will be:

Guava
Orange 
Pineapple

9. Create another fileColors.txt, add content (one color per line), and display the content?

To create and add content in Colors.txt we use 'vim' command:

vim Colors.txt

Add different colors name per line in file. To save and exit, press 'esc' then ':wq' and then 'enter'.

To display the content we use 'cat' command

cat Colors.txt

10. Add content inColors.txt(one in each line) - Red, Pink, White, Black, Blue, Orange, Purple, Grey. Then, prepend "Yellow" to the beginning of the file?

To add content in Colors.txt we use 'vim' command

vim Colors.txt

Add content in above file

Red
Pink
White
Black
Blue 
Orange
Purple
Grey

Save and exit by pressing 'esc', ':wq' and 'enter'.

To prepend "Yellow" in the beginning of the file, we can use 'echo' command with the append operator '>>' and a temporary file. Then, overwrite Colors.txt with the new content.

echo "Yellow" > tmp.txt
cat Colors.txt >> tmp.txt
mv tmp.txt Colors.txt

To display we use 'cat' command

cat Colors.txt

The output will be:

Yellow
Red
Pink
White
Black
Blue 
Orange
Purple
Grey

11. Find and display the lines that are common betweenfruits.txtandColors.txt?

First, ensure both 'fruits.txt' and 'Colors.txt' are sorted alphabetically.

We'll use 'sort' command to sort the content and -o flag to create new file with sorted output.

sort fruits.txt -o fruits_sorted.txt
sort Colors.txt -o Colors_sorted.txt

Now, use the 'comm' command with the sorted files to find and display the common lines (in this case, common fruits/colors):

comm -12 fruits_sorted.txt Colors_sorted.txt

• -1 suppress lines unique to fruits_sorted.txt.

• -2 suppress lines unique to Colors_sorted.txt.

• -12 display lines common to both files.

12. Count the number of lines, words, and characters in bothfruits.txtandColors.txt?

We used 'wc ' command to count the number of lines, words and characters in files.

For Fruits.txt:

wc fruits.txt

For Colors.txt:

wc Colors.txt

• The first column will represent: Number of lines.

• The second column will represent: Number of words.

• The third column will represent: Number of characters.

Conclusion

We explored fundamental Linux commands for efficient file operations like: Viewing File Content with Line Numbers, Changing File Permissions, Command History Management, Directory Deletion, Creating and Manipulating Text Files, Text Manipulation and File Metrics.

These foundational Linux commands empower developers and system administrators to efficiently manage file operations, ensure security, and analyze data effectively. Mastering these skills enhances proficiency and productivity in Linux environments, enabling smoother workflow and robust file management capabilities.

Happy Learning🌟