The fourth track in the 9th dialog systems technology challenge (DSTC9) requires participants to build a dialog system capable of task-oriented conversations in a co-observed multimodal context using the Situated Interactive Multimodal Conversations (SIMMC) dataset. The traditional approach to building such a dialog system is to fuse the multimodal context using an attention network. However, this approach makes it difficult to use pre-trained models such as BERT or GPT-2. In this paper, we present an end-to-end pipelined neural architecture for task-oriented dialog systems that successfully combines the multimodal context with conversational context. In the official evaluation metrics, our dialog system ranked first place on the response generation task (subtask 2-a) with a BLUE-4 score of 0.128, ranked first place in dialog state tracking task (subtask 3) with an F1 score of 79.1% for slot filling. Our system also ranked second place in the re-sponse retrieval task (subtask 2-b) with mean reciprocal rank (MRR) of 0.38.