Learning to Schedule in Parallel-Server Queues with Stochastic Bilinear Rewards

We consider the problem of scheduling in multi-class, parallel-server queuing systems with uncertain rewards from job-server assignments. In this scenario, jobs incur holding costs while awaiting completion, and job-server assignments yield observable stochastic rewards with unknown mean values. The mean rewards for job-server assignments are assumed to follow a bilinear model with respect to features that characterize jobs and servers. Our objective is to minimize regret by maximizing the cumulative reward of job-server assignments over a time horizon, while keeping the total job holding cost bounded to ensure the stability of the queueing system. This problem is motivated by applications requiring resource allocation in network systems. A central challenge is to control the tradeoff between reward maximization and fair allocation for the stability of the underlying queuing system (i.e., maximizing network throughput). To address this challenge, we propose a scheduling algorithm based on a weighted proportional fair criteria augmented with marginal costs for reward maximization, incorporating a bandit algorithm tailored for bilinear rewards. Our algorithm admits a regret--queue length tradeoff. For any fixed control parameter V>0, it ensures a uniform expected queue length and time-average holding-cost bounds. For a target horizon T, choosing V_T=Θ(\sqrt{IT}) at initialization yields \widetilde O((\sqrt I+d^2)\sqrt T+1/δ) regret. Under this regret-optimized tuning, the corresponding expected queue length and time-average holding-cost bounds remain uniform over the execution time and scales as O(\sqrt{IT}+1/δ) and O(\sqrt{IT}/δ), respectively.