Automated Fiber Placement (AFP) is an additive composite manufacturing technique, and a pressing challenge facing this technology is defect detection and repair. Manual defect inspection is time consuming, which led to the motivation to develop a rapid automatic method of inspection. This paper suggests a new automated inspection system based on convolutional neural networks and image segmentation tasks. This creates a pixel by pixel classification of the defects of the whole part scan. This process will allow for greater defect information extraction and faster processing times over previous systems, motivating rapid part inspection and analysis. Fine shape, height, and boundary detail can be generated through our system as opposed to a more coarse resolution demonstrated in other techniques. These scans are analyzed for defects, and then each defect is stored for export, or correlated to machine parameters or part design. The network is further improved through novel optimization techniques. New training instances can also be created with every new part scan by including the machine operator as a post inspection check on the accuracy of the system. Having a continuously adapting inspection system will increase accuracy for automated inspections, cutting down on false readings.