부산대학교 도서관

End-to-end learning has become a popular method to optimize a constellation shape of a communication system. When the channel model is differentiable, end-to-end learning can be applied with conventional backpropagation algorithm for optimization of the shape. A variety of optimization algorithms have also been developed for end-to-end learning over a non-differentiable channel model. In this paper, we compare a gradient-free optimization method based on the cubature Kalman filter, model-free optimization and backpropagation for end-to-end learning on a fiber-optic channel modeled by the split-step Fourier method. The results indicate that the gradient-free optimization algorithms provide a decent replacement to backpropagation in terms of performance at the expense of computational complexity. Furthermore, the quantization problem of finite bit resolution of the digital-to-analog and analog-to-digital converters is addressed and its impact on geometrically shaped constellations is analysed. Here, the results show that when optimizing a constellation with respect to mutual information, a minimum number of quantization levels is required to achieve shaping gain. For generalized mutual information, the gain is maintained throughout all of the considered quantization levels. Also, the results imply that the autoencoder can adapt the constellation size to the given channel conditions.