This paper introduces a novel hardware fabric implementing a stochastic neural network called the Neural Sampling Machine (NSM) using stochasticity in synaptic connections for approximate Bayesian inference. A hybrid stochastic synapse, pairing a ferroelectric field-effect transistor (FeFET)-based analog weight cell with a stochastic selector element, is demonstrated. Network simulations show the NSM's ability for autonomous weight normalization in continual online learning and Bayesian inferencing, achieving 98.25% accuracy on MNIST image classification and estimating prediction uncertainty.