# CSE 1110 Software Quality Testing — Complete Summary

---

## Lecture 1: Effective and Systematic Software Testing

### Key Concepts — Testing Principles

- **Exhaustive testing is impossible**: you cannot test every possible input; trade-offs are unavoidable
- **Absence-of-errors fallacy**: testing shows bugs are present, never that they are absent (Dijkstra, 1970)
- **Pesticide paradox**: repeating the same tests eventually stops finding new bugs; you need varied techniques and inputs
- **Defect clustering**: bugs are not evenly distributed; more bugs tend to appear in complex, heavily modified, or boundary areas
- **Testing is context-dependent**: the best approach depends on the specific situation; no single technique fits all
- **Verification is not validation**: verification = "having the system right" (does code match spec?); validation = "having the right system" (does it solve the user's problem?)
- **No matter what testing you do, it will never be perfect or enough**

### The Testing Pyramid

| Level | Scope | Characteristics |
|-------|-------|-----------------|
| **Unit tests** (bottom, most) | Single method/class | Fast, easy to write, full control over input/output, no mocks needed (ideally) |
| **Integration tests** (middle) | Multiple components interacting | More complex setup, tests communication between components (e.g., DAO + database) |
| **System tests** (top, fewest) | Entire system end-to-end | Most realistic, slowest, hardest to make deterministic, fewest in number |
| **Manual tests** (above system) | Exploration/validation | Not automated; useful for exploratory testing and gaining insight |

- The **testing trophy** is an alternative view: thinner bottom (unit), bigger middle (integration), thinner top (system). Common in microservices architectures and database-centric systems.
- Google classifies tests by **size** rather than scope: small (single process), medium (multiple processes/localhost), large (full end-to-end).

### Key Concepts — What Tests Do

- Tests help **find bugs**
- Tests give **confidence** that code works and will keep working
- Tests **communicate** how code is supposed to behave
- Tests serve as **starting point for debugging**
- Tests **show others how to use code**

### Method: Choose the Right Test Level

1. **Unit tests**: business logic, algorithms, single pieces of functionality that don't depend on external services
2. **Integration tests**: when component interacts with external services (database, web service, file system)
3. **System tests**: critical user journeys, risky parts of the system, after thorough unit and integration testing
4. Use **risk analysis** to prioritize what to test at system level

### WARNING

- Don't blindly aim for 100% of any coverage metric — understand what's uncovered and why
- The pesticide paradox means you should **vary your inputs and techniques**, not just keep adding more tests of the same kind
- Testing can be effective to show bugs are present, but never to show they are absent

---

## Lecture 2: Specification-Based Testing & Boundary Analysis

### Key Concepts

- **Specification-based testing** (domain testing): derive test cases from requirements, not from code
- Focus on the **specification**, not on the implementation
- **Equivalence partitioning**: divide inputs into partitions where all values in a partition behave similarly; test one representative from each partition
- **Boundary analysis**: bugs tend to happen at boundaries between partitions

### Method: The 7 Steps of Specification-Based Testing

1. **Understand the requirements** — identify business rules, inputs, outputs
2. **Explore the program** — ad hoc testing to build mental model (especially if you didn't write the code)
3. **Identify partitions** — for each input individually, for each input in combination with others, for the output
4. **Analyze the boundaries** — find on points and off points for conditions
5. **Devise test cases** — derive from the partitions and boundaries identified
6. **Automate the test cases** — write JUnit tests
7. **Augment with creativity and experience** — add edge cases, nulls, empty strings, etc.

### Boundary Testing: On Points and Off Points

- **On point** = the value that appears in the condition itself
  - If on point makes condition **false** → off point must make it **true**
  - If on point makes condition **true** → off point must make it **false**
- **Off point** = the nearest value to the on point that produces the opposite outcome

| Condition | On Point | Off Point(s) |
|-----------|----------|--------------|
| `value > 100` | 100 (condition false) | 101 (condition true) |
| `value >= 101` | 101 (condition true) | 100 (condition false) |
| `value == 100` | 100 (condition true) | 99 and 101 (condition false) |
| `value > n + 1` | n + 1 (condition false) | (n + 1) + 1 (condition true) |

- For **equality** (`==`), there are **two** off points (one below, one above)
- Be aware of **floating-point precision errors** — context matters for what counts as "on" the boundary
- **Loop boundary criterion**: test loop zero times, once, and more than once

### Partitioning Strategy

- Start with **each input individually**: null, empty, single element, multiple elements
- Then **inputs in combination**: common elements, no common elements, boundary sizes
- Then **output expectations**: non-null, null, empty collection, specific values
- Pick **reasonable** input values for dimensions you don't care about
- Go for the **simplest** input when in doubt
- Test for **nulls and exceptional cases**, but only when it makes sense

### Key Concepts — Parameterized Tests

- Use parameterized tests when tests have the same skeleton with different inputs
- Avoid redundant code clones
- If you need to add conditions to handle parameters differently, prefer separate tests for clarity

---

## Lecture 3: Structural Testing & Code Coverage

### Key Concepts

- **Structural testing** uses the **source code** to augment test suites derived from specification-based testing
- Goal: identify parts of code **not yet covered** and reflect on whether they should be
- Code coverage should **guide** testing, not be a number to blindly achieve

### Coverage Criteria (in order of strength)

| Criterion | What it checks | Relative cost |
|-----------|---------------|---------------|
| **Line (statement) coverage** | Every line is executed at least once | Lowest |
| **Branch (decision) coverage** | Every branch (true/false) of every decision is taken | Low |
| **Condition coverage** | Every atomic condition takes true and false | Medium |
| **Condition + branch coverage** | Every atomic condition and every branch outcome | Medium |
| **MC/DC** (Modified Condition/Decision Coverage) | Each condition independently affects the outcome | High |
| **Path coverage** | Every possible path through the code | Highest (often infeasible) |

### Subsumption Relations

- **MC/DC subsumes** condition + branch coverage
- **Condition + branch subsumes** branch coverage
- **Branch coverage subsumes** line coverage
- Path coverage subsumes everything (but is usually infeasible)

### Method: Achieving MC/DC

1. Build the **truth table** for the expression
2. For each condition, find a **pair of test cases** where:
   - Only that condition changes (true ↔ false)
   - The overall expression outcome changes
3. This is **unique-cause MC/DC**. If impossible (e.g., repeated condition), use **masked MC/DC** (fix all other variables)
4. **Theoretical lower bound**: N + 1 test cases for N conditions
5. Some expressions (like `A and (A or B)`) cannot achieve MC/DC — may indicate poorly designed code

### Method: Calculate Line Coverage

- Disregard method signature lines and closing braces
- Count executable lines covered ÷ total executable lines

### Method: Calculate Branch Coverage

- `branch coverage = branches covered / total branches × 100%`
- An `if-else` has 2 branches; an `if` without `else` has 2 branches (true and false paths)
- For `if (x || y)`: 2 branches (true/false) + 4 conditions (x true, x false, y true, y false)

### What Should Not Be Covered?

- Trivial code like `equals()`, `hashCode()`, simple getters/setters
- Checked exceptions wrapped to unchecked (e.g., try-catch wrapping `URISyntaxException`)
- Code tested implicitly by other tests
- Start from "everything should be covered" and make exceptions with justification

### Key Concepts — Mutation Testing

- **Mutants** = modified versions of the program with simple bugs injected
- **Killed mutant**: test suite fails → test is good
- **Surviving mutant**: test suite passes → test is weak or assertion is missing
- **Competent programmer hypothesis**: real bugs are simple variations of the correct code
- **Coupling effect**: if tests catch simple bugs, they catch complex ones too
- **Pitest** is the main mutation testing tool for Java

Common mutators in Pitest:
- Conditionals boundary: `<` → `<=`, etc.
- Increment: `i++` → `i--`
- Math operators: `+` → `-`, etc.
- True returns: replace boolean with `true`
- Remove conditionals: replace `if` with `if (true)`

### WARNING

- Coverage alone is **not enough**: you can achieve 100% branch coverage with weak test cases
- Structural testing must be **combined with specification-based testing**
- Don't let coverage tools dictate your test cases; use them to find what you've missed

---

## Lecture 4: Designing Contracts & Design by Contract

### Key Concepts

- **Pre-conditions**: what the method requires as input (caller's responsibility)
- **Post-conditions**: what the method guarantees about its output (callee's responsibility)
- **Invariants**: conditions that always hold for an object's state (e.g., a list is never null, a basket's total is always correct)

### Implementation Approaches

- **Manual if-checks** with `throw new RuntimeException("...")`
- **`assert` keyword**: `assert value >= 0 : "Value cannot be negative."` → throws `AssertionError` if disabled
- **Javadoc**: document pre/post conditions in `@param` and `@return` descriptions
- **Contract enforcement**: pre-condition violations → caller gets exception; post-condition violations → implementation has a bug

### Liskov Substitution Principle & Contract Changes

- **Subclass can weaken pre-conditions** (accept more inputs) — caller is guaranteed to provide the more restrictive pre-condition, so it works
- **Subclass can strengthen post-conditions** (promise more) — caller expects the weaker guarantee, so receiving more is fine
- **Subclass must NOT strengthen pre-conditions or weaken post-conditions** — breaks LSP

| Change | Valid? | Why |
|--------|--------|-----|
| Weaken pre-condition | Yes | More inputs accepted, existing callers still satisfy weaker requirement |
| Strengthen post-condition | Yes | Caller gets more than expected |
| Strengthen pre-condition | No | Existing callers may not satisfy stricter requirement |
| Weaken post-condition | No | Caller may depend on the stronger guarantee |

### Assert vs Exceptions

- **Asserts**: for developer errors, internal consistency checks, contracts; can be disabled
- **Exceptions**: for invalid inputs from external sources, defensive programming
- Use **asserts** for pre/post conditions and invariants in development
- Use **exceptions** for input validation from external/untrusted sources

### When Not to Use Design by Contract

- When contracts are constantly changing (instability)
- When the overhead outweighs the benefits
- For very simple methods where contracts are obvious
- In performance-critical paths (asserts can be disabled, but explicit checks add overhead)

---

## Lecture 5: Property-Based Testing

### Key Concepts

- **Example-based testing**: write specific input-output pairs
- **Property-based testing**: define a **property** (a rule that must always hold), generate random inputs, check property holds for all
- Tools: **jqwik** (Java), **QuickCheck** (Haskell), **Hypothesis** (Python)

### When to Use Property-Based Testing

- Properties that hold for **all inputs** (not just specific cases)
- Inverting known operations
- Commutativity, idempotency, round-trip properties
- Type-invariants (generated values match their type)
- When you want to explore more inputs than you could manually

### Common Property Types

| Property | Example |
|----------|---------|
| **Round-trip** | `parse(serialize(x)) == x` |
| **Commutativity** | `add(a, b) == add(b, a)` |
| **Idempotency** | `sort(sort(x)) == sort(x)` |
| **Type-invariant** | Generated integers are within valid range |
| **Inverse** | `sqrt(x)^2 == x` (for x >= 0) |

### Method: Write a Property-Based Test

1. Identify a **property** that should hold for all inputs
2. Declare **generators** (how to produce random inputs)
3. Write an **assertion** that checks the property
4. Run with many generated inputs

### Common Issues

- **Shrinking**: tools try to find minimal failing inputs — useful for debugging
- **Constraints**: use `.where()` to filter invalid generated values
- **Creativity is key**: the quality of property-based tests depends on finding meaningful properties
- Don't test implementation details; test **behavioral properties**

### Example: Testing a unique() Method

```java
@Property
boolean uniquePreservesSize(@ForAll("list") List<Integer> list) {
    List<Integer> unique = list.stream().distinct().toList();
    // Property: no duplicates in result
    return unique.size() == unique.stream().distinct().count();
}
```

### WARNING

- Property-based testing **complements** (doesn't replace) example-based testing
- You still need specific test cases for known bug-prone scenarios
- Be creative in finding properties — that's where the value lies

---

## Lecture 6: Test Doubles and Mock Objects

### Key Concepts

Test doubles replace real dependencies during testing. Five types (from xunitpatterns.com):

| Type | Purpose | Example |
|------|---------|---------|
| **Dummy** | Fills parameter list, never used | `new User(null)` |
| **Fake** | Working implementation but simplified | `HashMap` as fake `Map`; in-memory database |
| **Stub** | Returns predetermined values | `when(service.getUser()).thenReturn(user)` |
| **Spy** | Records how it was called | `verify(repo).save(any())` |
| **Mock** | Has expectations about how it will be called | `mockService.verifyMethodCall()` |

### When to Mock vs When Not to

**Mock when**:
- External service (database, web API) — use stub or fake instead
- Complex dependency that's hard to set up
- Need to verify interactions (method was called with correct args)

**Don't mock when**:
- Simple value objects
- Code you own and can test directly
- When mocking makes tests brittle (fragile to refactoring)
- Fake objects often suffice (e.g., in-memory database)

### Stubbing and Mocking with Mockito

```java
// Stub: return a value
when(repo.findById(1)).thenReturn(optionalUser);

// Mock: verify interactions
verify(repo).save(any(User.class));
verify(repo, never()).delete(any());

// Capture arguments
ArgumentCaptor<User> captor = ArgumentCaptor.forClass(User.class);
verify(repo).save(captor.capture());
assertThat(captor.getValue().getName()).isEqualTo("John");

// Simulate exceptions
when(service.process()).thenThrow(new RuntimeException("fail"));
```

### Design Patterns for Testability

- **Dependency Injection**: pass dependencies via constructor
- **Ports and Adapters**: separate domain from infrastructure
- **Interface segregation**: small interfaces are easier to mock

### Date/Time Wrappers

- Mocking `LocalDateTime.now()` is hard
- Use **dependency injection** to pass a `Supplier<LocalDateTime>` or `Clock`

---

## Lecture 7: Designing for Testability

### Key Concepts

- Testable code is **well-designed code** — the two go hand in hand
- Separating infrastructure from domain code is fundamental

### Method: Make Code Testable

1. **Separate infrastructure from domain** — database calls, file I/O, network calls belong in infrastructure layer
2. **Dependency Injection** — inject dependencies via constructors, not `new` inside methods
3. **Make classes observable** — add methods/properties to verify state or behavior
4. **Keep methods focused** — single responsibility makes testing easier

### Dependency Injection vs Method Parameters

| Approach | Use When |
|----------|----------|
| **Constructor injection** | Long-lived dependency (repository, service) |
| **Method parameter** | Short-lived value (configuration, data) |

### Code Smells for Untestable Code

- **Static methods** — hard to mock, hard to substitute
- **Singletons** — global state, hard to isolate
- **Private methods** — don't test directly; test through the public API
- **Tight coupling** — class creates its own dependencies with `new`
- **God classes** — too many responsibilities

### Hexagonal Architecture / Ports and Adapters

- Domain logic sits at the center (pure, no dependencies)
- Infrastructure (database, web, file system) is on the outside (adapters)
- Interfaces (ports) define the contracts between layers
- Makes unit testing trivial: just implement the port with a stub/fake

### Method: Refactor to Achieve Better Coverage

1. Identify **unreachable branches** — add test cases from specification
2. If still unreachable, check if the code is **truly unreachable** (dead code)
3. If the dependency makes testing impossible, apply **dependency injection**
4. If the class has too many responsibilities, **split it**

---

## Lecture 8: Test-Driven Development (TDD)

### Key Concepts

- **TDD cycle** (Red-Green-Refactor):
  1. **Red**: Write a failing test for the next bit of functionality
  2. **Green**: Write just enough code to make the test pass
  3. **Refactor**: Clean up the code (both production and test) while keeping tests green

### TDD Benefits

- Tests are written **before** code — nothing goes untested
- Drives **simple design** — only implement what's needed
- Acts as **living documentation**
- Encourages **small, focused methods**
- Makes **refactoring safe**

### To TDD or Not to TDD?

- **TDD is not 100% of the time** — it depends on context
- Good for: new functionality, complex algorithms, well-understood requirements
- Less suitable for: exploration/probing, UI prototyping, when requirements are unclear
- **Research**: studies show TDD can improve code quality but has a learning curve

### TDD Variants

- **London school (strict TDD)**: no production code until test fails; tests only test production code; no testing frameworks in production code
- **Chicago school**: more relaxed; allows testing frameworks in production; focus on the red-green cycle
- **Outside-in TDD**: start with system-level tests and work inward
- **Inside-out TDD**: start with domain objects and work outward

### TDD and Proper Testing

- TDD naturally leads to **good test practices**: single responsibility, strong assertions, no duplication
- The refactoring step is crucial — it prevents test and production code from becoming messy
- TDD encourages **dependency injection** naturally (you need to be able to inject test doubles)

### WARNING

- TDD is a **design technique** first, a testing technique second
- Don't use TDD as an excuse to skip specification-based or structural testing
- The goal is not just to pass tests but to **design good software**

---

## Lecture 9: Writing Larger Tests (SQL & Web Testing)

### Key Concepts

- **Integration tests**: test components interacting with external systems (databases, web services)
- **System tests**: test the entire application end-to-end
- Same testing techniques apply (specification-based, boundary, structural) but at a larger scale

### SQL Testing

**What to test in SQL queries**:
- Correct results for happy paths
- Edge cases: empty results, null values, duplicate data
- Performance: very large datasets
- Error handling: invalid inputs

**Best practices**:
- Use an **in-memory database** (H2, HSQLDB) for fast, isolated tests
- **Open and commit transactions** — ensure test data is properly managed
- **Helper methods** to reduce test boilerplate
- **Test data builders** for creating realistic test data
- **Use good assertion APIs** (AssertJ) for readable assertions
- **Minimize required data** — don't set up more DB state than needed
- **Consider schema evolution** — tests should work with DB migrations
- **Take schema evolution into consideration** when writing tests

### Web Testing with Selenium / JavaScript Frontend

**Key principles**:

1. **Provide a way to set the system to the state the web test requires** — use APIs or admin panels to set up state
2. **Each test runs in a clean environment** — no shared state between tests
3. **Give meaningful names to HTML elements** — use `data-test-id`, `id`, `name` attributes for reliable selectors
4. **Visit every step of a journey only when under test** — don't test navigation that's not relevant
5. **Assertions should use data from the POS** (Point of Sale / actual system output), not hardcoded values
6. **Pass important configurations to the test suite** — environment URLs, credentials as parameters
7. **Use the Page Object pattern** — encapsulate UI interactions in page classes

### Page Object Pattern

```java
class LoginPage {
    private WebDriver driver;

    public LoginPage(WebDriver driver) {
        this.driver = driver;
    }

    public DashboardPage login(String user, String pass) {
        driver.findElement(By.id("username")).sendKeys(user);
        driver.findElement(By.id("password")).sendKeys(pass);
        driver.findElement(By.id("login")).click();
        return new DashboardPage(driver);
    }
}
```

### System Test Flakiness Causes

- Non-deterministic components (timers, random values, async operations)
- Race conditions
- Shared state between tests
- External dependencies that are unreliable

### WARNING

- Web/system tests are **slow and fragile** — keep them to a minimum
- Use them for **critical user journeys** only
- Prioritize unit and integration tests for breadth
- Database tests should use **in-memory databases** for speed and isolation

---

## Lecture 10: Test Code Quality

### Principles of Maintainable Test Code

| Principle | Description |
|-----------|-------------|
| **Tests should be fast** | Slow tests discourage running them frequently |
| **Tests should be cohesive, independent, and isolated** | Each test tests one thing; no shared state |
| **Tests should have a reason to exist** | Not just for coverage — each test should verify behavior |
| **Tests should be repeatable and not flaky** | Same result every time, regardless of environment |
| **Tests should have strong assertions** | Assert the important behavior, not implementation details |
| **Tests should break if the behavior changes** | If a test passes despite a behavior change, it's weak |
| **Tests should have a single and clear reason to fail** | One assertion failure per test makes debugging easy |
| **Tests should be easy to write** | Complex test setup indicates design problems |
| **Tests should be easy to read** | Test code is documentation |
| **Tests should be easy to change and evolve** | Follow DRY, use helper methods, avoid duplication |

### Test Smells

| Smell | Description | Fix |
|-------|-------------|-----|
| **Excessive duplication** | Same setup/assertions repeated across tests | Extract helper methods, use parameterized tests |
| **Unclear assertions** | Assertion message doesn't explain what's being verified | Add meaningful assertion messages |
| **Bad handling of complex/external resources** | Tests that depend on live DB, network, files | Use test doubles, in-memory alternatives |
| **Fixtures that are too general** | One huge setup used by many tests with different needs | Create focused fixtures for specific scenarios |
| **Sensitive assertions** | Assertions depend on ordering, timestamps, or other unstable values | Assert on relevant properties, not exact values |

### WARNING

- Test code quality matters as much as production code quality
- **Refactor tests** just like production code
- **Dead test code** (unused tests) is worse than no tests — it misleads
- Don't test **private methods** directly — test through the public API

---

## Exam Structure & Topics

### Exam Format

- **WebLab exam**, on campus, with optional IntelliJ support
- Mostly **programming**, some theory (MCQ)
- Automatically graded by **Andy** (the assessment tool)
- **Code must compile and tests must pass** — otherwise zero points
- Documentation available: Java, JUnit, AssertJ, Mockito, Selenium, jqwik

### Andy's Assessment Criteria

| Criterion | Description | Typical Weight |
|-----------|-------------|----------------|
| **Branch coverage** | Percentage of branches covered | ~10% |
| **Mutation coverage** | Percentage of mutants killed | ~30% |
| **Code checks** | Static analysis for test quality (naming, structure) | ~8% |
| **Meta tests** | Manually written tests that student tests must kill | ~60% |

### Common Exam Question Types

1. **Testing pyramid** — identify appropriate test levels
2. **Testing principles** — MCQ on pesticide paradox, defect clustering, exhaustive testing, verification vs validation
3. **Invariant in subclass** — LSP and contract changes
4. **Test-driven development** — identify TDD steps, benefits, limitations
5. **Test code quality** — identify test smells, apply principles
6. **Branch coverage** — calculate coverage, create tests to achieve coverage
7. **MC/DC** — create test suite for complex boolean expression
8. **Boundary testing** — identify on/off points
9. **Mock objects** — when to mock, how to use
10. **Design-by-contract** — pre/post conditions, asserts
11. **Code coverage** — line vs branch vs condition coverage
12. **Mutation testing** — create mutants that are killed vs survive
13. **Specification-based testing** — derive test cases from requirements
14. **Property-based testing** — identify properties, write jqwik tests
15. **SQL testing** — write tests for database operations
16. **Web testing** — Selenium tests, page objects
17. **Fuzzing** — random input testing
18. **Stubs vs mocks vs dummies** — identify test double types

### Andy: Automatic Grading Tool

- Andy assesses student test suites on multiple criteria
- Provides feedback on coverage, mutation testing, code quality, and meta tests
- Used as the assessment tool for CSE1110
- Available at the TU Delft GitLab

---

## Quick Reference: When to Use What

| Problem Type | Technique |
|-------------|-----------|
| Derive test cases from requirements | Specification-based testing (7 steps) |
| Find boundary values | On/off point analysis |
| Ensure all code paths tested | Structural testing (branch, condition, MC/DC) |
| Complex boolean expressions | MC/DC (N+1 test cases) |
| Test strong assertions | Mutation testing (kill all mutants) |
| Test external dependencies | Test doubles (stubs, mocks, fakes) |
| Test without knowing implementation | Property-based testing |
| Guide development | TDD (Red-Green-Refactor) |
| Test database operations | Integration tests with in-memory DB |
| Test entire user journey | System tests with Selenium |
| Write maintainable tests | Follow test code quality principles |
| Test contract inheritance | LSP: weaken pre, strengthen post |
| Make code testable | Dependency injection, separate infrastructure |