How to Replace Strings in Python using re.sub

Avatar

By squashlabs, Last Updated: June 8, 2023

How to Replace Strings in Python using re.sub

A. Breaking Down the re.sub Syntax

The re.sub function’s syntax is as follows:

re.sub(pattern, repl, string, count=0, flags=0)

The function takes a pattern that it looks for in the provided string. Once located, it replaces the pattern with the repl argument. The count argument defines how many occurrences of the pattern are replaced, with the default being all occurrences (0). The flags argument can modify the pattern matching, for instance making it case-insensitive.

Related Article: String Comparison in Python: Best Practices and Techniques

B. Fundamentals: Pattern, Replacement, and String

Pattern: The pattern is a string containing a regular expression, also known as a regex, that you’re searching for in the string.

Replacement: The replacement is the string that you’d like to replace the pattern with. This can also be a function, allowing for dynamic replacement logic.

String: The string is the text within which you’re making substitutions. It’s the “search space” for the pattern.

For example, if we want to replace all occurrences of ‘python’ with ‘snake’ in a string, we would do:

import re
text = "I love python. Python is my favorite language."
new_text = re.sub('python', 'snake', text, flags=re.IGNORECASE)
print(new_text) # Outputs: "I love snake. Snake is my favorite language."

Here, ‘python’ is our pattern, ‘snake’ is our replacement, and text is our string. The re.IGNORECASE flag ensures the function is case-insensitive.

II. String Replacement with re.sub

In this section, we will look at how re.sub function can be used in different scenarios.

A. Simple Text Replacement

Let’s consider a simple example where we replace occurrences of one word with another:

import re
sentence = "The cat sat on the mat."
new_sentence = re.sub('cat', 'dog', sentence)
print(new_sentence) # Outputs: "The dog sat on the mat."

Here, we replaced all occurrences of ‘cat’ with ‘dog’.

Related Article: How To Limit Floats To Two Decimal Points In Python

B. Replacement with Regex Patterns

Now, let’s use a regular expression pattern to replace text:

import re
sentence = "The prices are $100, $200 and $300."
new_sentence = re.sub('\$\d+', 'price', sentence)
print(new_sentence) # Outputs: "The prices are price, price and price."

We used a regular expression to match dollar amounts and replaced each with the word ‘price’.

C. Substituting Special Sequences

We can also replace special sequences like \d (any digit) or \w (any alphanumeric character):

import re
sentence = "Username: user123, Password: pass456"
new_sentence = re.sub('\w+:\s\w+', '[REDACTED]', sentence)
print(new_sentence) # Outputs: "[REDACTED], [REDACTED]"

In this example, we redacted the sensitive information from the sentence.

III. Additional Features

Related Article: How To Rename A File With Python

A. Count Parameter in Depth

The count parameter limits the number of substitutions made. Let’s see an example:

import re
text = "apple, apple, apple"
new_text = re.sub('apple', 'orange', text, count=2)
print(new_text) # Outputs: "orange, orange, apple"

In the above case, we limit the replacement of ‘apple’ to only the first two occurrences.

B. The repl Function Parameter

The repl parameter can also be a function that takes a match object and returns a string. This allows for dynamic replacements:

import re
def reverse_match(match):
    return match.group(0)[::-1]

text = "123 abc 456 def"
new_text = re.sub('\w+', reverse_match, text)
print(new_text) # Outputs: "321 cba 654 fed"

Here, we used a function to reverse each word in the string. The group(0) method of the match object returns the full match.

IV. Advanced Text Manipulation with re.sub

Python’s re.sub offers various advanced features that allow complex text manipulations.

Related Article: How To Check If List Is Empty In Python

A. Multi-step Text Processing

At times, multiple re.sub operations can be chained for intricate text processing. Consider this:

import re
text = "HELLO, how ARE YOU?"
text = re.sub('[A-Z]+', lambda m: m.group(0).lower(), text) # Lowercasing
text = re.sub('\w+', lambda m: m.group(0).capitalize(), text) # Capitalizing words
print(text) # Outputs: "Hello, How Are You?"

B. Handling Complex Patterns

Regular expressions support advanced constructs like non-capturing groups, lookaheads, and lookbehinds.

import re
text = "100 cats, 200 dogs, 300 birds."
new_text = re.sub('(\d+)\s(?=dogs)', '150', text)
print(new_text) # Outputs: "100 cats, 150 dogs, 300 birds."

Here, (?=dogs) is a lookahead assertion ensuring replacements only when ‘dogs’ follows the number.

C. Applying Lookahead and Lookbehind Assertions

Advanced regex features like lookaheads and lookbehinds can be used with re.sub for complex pattern recognition:

import re
text = "Add 100, minus 100, add 200, minus 200."
new_text = re.sub('(?<=add\s)\d+', '50', text, flags=re.IGNORECASE)
print(new_text) # Outputs: "Add 50, minus 100, add 50, minus 200."

Here, (?<=add\s) is a positive lookbehind assertion, which matches ‘add’ followed by a space but doesn’t include it in the match. Thus, only numbers following ‘add’ get replaced.

Related Article: How To Check If a File Exists In Python

V. Real-World Applications of re.sub

The re.sub function can be an invaluable tool in a variety of practical scenarios.

A. Data Cleaning in Pandas DataFrames

When working with Pandas DataFrames, re.sub can be applied to clean up data:

import re
import pandas as pd

data = {'text': ['Hello!!', 'Python...', '#Regular_Expressions']}
df = pd.DataFrame(data)

df['text'] = df['text'].apply(lambda x: re.sub('[^a-zA-Z\s]', '', x))
print(df)

This snippet removes all non-alphabet characters from the DataFrame’s ‘text’ column.

B. Text Normalization for Natural Language Processing

re.sub can also be used to normalize text in natural language processing tasks:

import re
text = "I'll be there at 4pm!!"

# Lowercasing and removing non-word characters
text_normalized = re.sub('[^a-z\s]', '', text.lower())
print(text_normalized) # Outputs: "ill be there at pm"

Related Article: How to Use Inline If Statements for Print in Python

C. Web Scraping and Information Extraction

When extracting information from web pages, re.sub can help clean the scraped data:

import re
scraped_data = "Hello, World!"

# Removing HTML tags
clean_data = re.sub('<.*?>', '', scraped_data)
print(clean_data) # Outputs: "Hello, World!"

D. Log Files Processing

re.sub is useful in processing log files, for example to anonymize sensitive data:

import re
log_line = "INFO - User john_doe accessed the system."

# Anonymizing usernames
anonymized_log = re.sub('User \w+', 'User [REDACTED]', log_line)
print(anonymized_log) # Outputs: "INFO - User [REDACTED] accessed the system."

VI. Beyond The Substitution: Optimizing re.sub Usage

Beyond basic and advanced usage, optimizing re.sub can bring substantial performance benefits, especially with large-scale data.

Related Article: How to Use Stripchar on a String in Python

A. Precompiled Patterns for Performance

Precompiling regex patterns with re.compile can improve performance when using the same pattern multiple times:

import re
text = "abc 123 def 456 ghi 789"
pattern = re.compile('\d+')

# Using the compiled pattern
new_text = pattern.sub('number', text)
print(new_text) # Outputs: "abc number def number ghi number"

B. Handling Unicode Characters

re.sub can also handle unicode characters, essential when dealing with non-English text or special symbols:

import re
text = "Mëtàl Hëàd 🤘"
new_text = re.sub('ë', 'e', text)
print(new_text) # Outputs: "Metal Head 🤘"

In this case, we replaced all occurrences of ‘ë’ with ‘e’.

VII. String Replacement: Practical Examples

Let’s explore some real-world, practical examples to solidify the understanding of re.sub.

Related Article: How To Delete A File Or Folder In Python

A. Handling Date and Time Strings

re.sub is useful when dealing with dates in different formats:

import re
date = "Today's date is 12-31-2023"
new_date = re.sub('(\d{2})-(\d{2})-(\d{4})', r'\2/\1/\3', date)
print(new_date) # Outputs: "Today's date is 31/12/2023"

Here, we rearranged the date format from MM-DD-YYYY to DD/MM/YYYY.

B. Extracting Information from Log Files

Extracting information from log files becomes easy with re.sub:

import re
log_line = "[2023-06-23 12:00:00] - ERROR - File not found: test.txt"

# Extracting file name
file_name = re.sub('.*File not found: (\w+\.\w+).*', r'\1', log_line)
print(file_name) # Outputs: "test.txt"

C. Implementing Text Censorship

re.sub can help in implementing a simple text censorship system:

import re
text = "This is a secret message."
censored_text = re.sub('secret', '******', text)
print(censored_text) # Outputs: "This is a ****** message."

In this case, we replaced the word ‘secret’ with asterisks.

Related Article: How To Move A File In Python

VIII. Substitutions

We have journeyed through a comprehensive exploration of Python’s re.sub. This final section will provide additional resources and tips to continue mastering this versatile function.

A. Mastering Regular Expressions

re.sub depends largely on the regex patterns used. To become a re.sub expert, consider mastering regular expressions. Resources like Regex101 provide interactive environments to learn and test regular expressions.

B. Python’s re Module

Apart from re.sub, the re module offers many other functions like re.search, re.match, and re.findall. Exploring these can open up new possibilities for text processing in Python.

Related Article: How to Implement a Python Foreach Equivalent

C. Text Processing Libraries

For more complex text processing tasks, libraries like NLTK, Spacy, and TextBlob can be valuable. They offer advanced functionalities like tokenization, part-of-speech tagging, and named entity recognition, which often incorporate regular expressions under the hood.

D. Real-World Projects

Applying re.sub in real-world projects is the best way to hone your skills. Whether it’s cleaning up a dataset, extracting information from logs, or automating edits in a large text file, real-world applications offer the best practice.

More Articles from the Python Tutorial: From Basics to Advanced Concepts series:

How to Use Slicing in Python And Extract a Portion of a List

Slicing operations in Python allow you to manipulate data efficiently. This article provides a simple guide on using slicing, covering the syntax, positive and negative... read more

How to Check a Variable’s Type in Python

Determining the type of a variable in Python is a fundamental task for any programmer. This article provides a guide on how to check a variable's type using the... read more

How to Use Increment and Decrement Operators in Python

This article provides a guide on the behavior of increment and decrement operators in Python. It covers topics such as using the += and -= operators, using the ++ and --... read more

How to Import Other Python Files in Your Code

Simple instructions for importing Python files to reuse code in your projects. This article covers importing a Python module, importing a Python file as a script,... read more

How to Use Named Tuples in Python

Named tuples are a useful feature in Python programming that allows you to create lightweight, immutable data structures. This article provides a simple guide on how to... read more

How to Work with CSV Files in Python: An Advanced Guide

Processing CSV files in Python has never been easier. In this advanced guide, we will transform the way you work with CSV files. From basic data manipulation techniques... read more