Testing objectives — Python: Automated testing

• How automatic tests work
• How tests are written
• How many checks do you need to write?
• How to check input data
• Conclusion

What is the most critical thing that tests should do? It is an essential question. The answer will help you understand how to write tests. It is the question we will answer in this lesson.

Imagine you've written a function called capitalize(text) that capitalizes the first letter of the string you pass:

capitalize('hello') # 'Hello'

Here's one way to implement it:

# main.py
def capitalize(text):
    first_char = text[0].upper()
    rest_substring = text[1:]
    return f'{first_char}{rest_substring}'

We've created a function, so we need to check that it works.

One way to do this is to open a REPL and call a function with different arguments:

python -i main.py
capitalize('hello, hexlet!')
# => 'Hello, hexlet!'

We have ensured that the function works — at least for the arguments we passed. If the test finds any errors, we should fix them and repeat the process.

This whole process is manual testing. Its purpose is to make sure that the code works as it should. And it makes no difference to us how we implement that function. It is the answer to the question posed at the beginning of this lesson: Tests verify that the code or application works correctly, and it doesn't matter how we write the code.

How automatic tests work

All we need to do for automated testing is to repeat the checks we performed during manual testing. Good old if and exceptions will do the trick. Even if you're not familiar with exceptions, that's okay. Two things are enough for this course: what exceptions are for and what their syntax is.

So far in Hexlet's courses, you've encountered bugs that occur accidentally, like calling a non-existent function and accessing a non-existent constant. But errors can also be created intentionally using exceptions, which we need. The exceptions mechanism generates errors:

# We throw a new exception
# The code after this expression will not run
# The script will stop with an error
raise Exception('Boom! An error occurred, stopping the execution')
print('nothing'); # It will never run

# After running, we will get the following output:
# The most recent calls are the last in the traceback:
#   File "main.py", line 8, in <module>
#     raise Exception('Boom! An error occurred, stopping the execution');
# Exception: Boom! An error occurred, stopping the execution

Now let's look at an example of the test:

# If the result of the function is not equal to the expected value
if capitalize('hello') != 'Hello':
    # We throw an exception and terminate the test execution
    raise Exception('The function is not working correctly!')

From the example above, we can see that tests have the same code as everything else. The code works in the same environment and is subject to the same coding rules and standards. It may also contain bugs, but that doesn't mean you have to write tests for tests. It's impossible to avoid all bugs, and you don't have to, or you won't get the resources you need for development.

We usually store tests in a directory at the root of the project. It's called tests, although there are other options:

    package-name/
    └── __init__.py
    tests/
    └── test_something.py

The structure of this directory mirrors the source code structure. For example, if our function capitalize(text) is defined in the file package-name/capitalize.py, it's better to put its test in the file tests/test_capitalize.py.

Imagine we want to change the code associated with this function. In this case, it's essential not to forget to run tests:

python tests/test_capitalize.py
# If everything is ok, the code will run silently
# If there is an error, an error message will be displayed

How tests are written

Developers import the tested functions themselves, call them with the necessary arguments, and verify that the functions return the expected values. Imagine that we edited the code and changed the function signature — the input or output parameter and its name. In this case, you would need to rewrite the tests.

Let us look at another example now. If the signature remains the same, but the internal parts of the function have changed, the tests should continue to run unchanged. It is what happens in this code:

def capitalize(text):
    first_char, *rest_chars = text
    rest_substring = ''.join(rest_chars)
    return f'{first_char.upper()}{rest_substring}'

Good tests know nothing about the internal structure of the code we test. It makes them more versatile and reliable.

How many checks do you need to write?

It's impossible to write tests that guarantee the code will work 100% of the time. That would require checking every possible argument, and that's physically impossible. On the other hand, without tests, there is no guarantee that the program will work at all.

When working with tests, you should focus on the variety of input data because any given function won't have too many fundamental use cases. For example, you can take any word and write one check in the case of capitalize(). Such a test will cover most of the possible scenarios.

Next, we look at borderline cases — situations in which the code can behave unexpectedly:

• Working with an empty string
• Processing None
• Dividing by zero that causes errors in most languages
• Specific situations for specific algorithms

For the capitalize() function, the borderline case will be an empty string:

if capitalize('') != '':
  raise Exception('The function is not working correctly!')

By adding a test on an empty string, we can see that calling the capitalize() function will cause an error. Within it, we access the first index of the string without checking whether it exists. It is how we should correct it:

def capitalize(text):
    if text == '':
        return ''
    first_char = text[0].upper()
    rest_substring = text[1:]
    return f'{first_char}{rest_substring}'

In many situations, borderline cases require separate processing and conditional constructions. We should make tests the way that they affect every one of them.

Don't forget that conditional constructions can generate non-obvious connections. For example, two independent conditional blocks generate four possible scenarios:

• The function did not trigger a conditional block to run
• The function triggered only the first conditional block to run
• The function triggered only the second conditional block to run
• The function triggered both conditional blocks to run

We refer to the combination of all possible behaviors of as cyclomatic complexity. It is a number that shows all possible code paths inside the function. Cyclomatic complexity is a good guideline for understanding what tests we should write and how many of them we should write.

Sometimes, borderline cases aren't related to conditional constructions. It's common for these situations to occur where there are computations of the boundaries of words or arrays. This code will work in most situations and fail in only a few:

# The code will return the element to the specified index
# We will get the default value if the index does not exist
def get_by_index(elements, index, default):
  return elements[index] if index <= len(elements) else default
# There is a bug because we used `<=` instead of `<`
# That's why this code sometimes works incorrectly

# The code works correctly
get_by_index(['zero', 'one'], 1, 'value') # 'one'
# The code works incorrectly
get_by_index(['zero', 'one'], 2, 'value') # Exception!

How to check input data

Let's look at input data type errors. For example, we can pass a number to the capitalize() instead of a string. How should the function behave in this case? Do we need to write a test for this?

Here's another interesting question. Is it necessary to handle these situations within capitalize()? You shouldn't because it's of little use. There should still be tests to check that the system works as a whole, and they will usually catch code problems at lower levels.

Passing the correct data to capitalize() does not lie with the function but the code calling it. And if we have tested it well, you will either catch this error or you won't face it at all.

But even if we handle the error within the function, there is no need to try to write tests that cover every bug. This results in many tests that require support and time to write. You need to be able to stop in time and move on to cover another part of your code.

Conclusion

In this lesson, we got to know automatic tests better. We ended up with the following directory structure:

    package-name/
    └── capitalize.py
    tests/
    └── test_capitalize.py

The test will look like this:

from capitalize import capitalize

if capitalize('hello') != 'Hello':
    raise Exception('The function is not working correctly!')

if capitalize('') != '':
    raise Exception('The function is not working correctly!')

print('All tests are passed!')

This test runs with the following command:

PYTHONPATH=package-name python tests/test_capitalize.py

If we wrote everything correctly, the test run will end with the line "All tests passed!" being output. If there is an error in the tests or the code, it will trigger an exception, and we'll get a corresponding message.