Artificial intelligence (AI) has gone a long way in improving our daily lives in many aspects. Neural networks (NNs) constitute a key component of almost every modern AI system. However, most of the current neural networks are black boxes representing functions from one high-dimensional space to another, but providing no justification or explanation for a given execution. For users to trust these systems, a neural network should therefore be able to highlight the patterns in the input data that it actually learned, or more generally, provide an explanation of how it arrived at its conclusion. My research aims to achieve this objective by interpreting neural networks as graphs; this can possibly let users know which parts of the neural network perform which task.