Braden Holtby made 29 saves to help the Capitals avoid back-to-back losses for the first time this year after falling 3-1 to Pittsburgh on Wednesday.
Sergei Bobrovsky stopped 26 shots for the Jackets.
Williams scored his second goal of the season to give Washington the lead at 14:16 in the second, taking Marcus Johansson's pass and firing a one-timer through traffic, over Bobrovsky's left shoulder and into the top right corner.
After Washington killed two penalties, Oshie's fourth goal made it 2-0 at 11:59 in the third when the Capitals finished a 3-on-2 breakaway.
Alex Ovechkin carried the puck down the left and passed across to Nicklas Backstrom, who tapped it back for Oshie to fire a one-time slap shot into the top right corner.
Matt Calvert scored less than three minutes later for Columbus, which fell to 2-2-0 under new coach John Tortorella. He lifted a wrist shot over Holtby on a rebound.
Capitals coach Barry Trotz challenged the goal, appearing to claim goalie interference on center Boone Jenner, but it stood.
The Blue Jackets began brightly with 10 shots in the first period, one more than they had over the first two periods of Tuesday's 3-1 win at New Jersey.
The Capitals still had the best chance of the first on a late power play, with Bobrovsky denying Oshie's rebound from in close after stopping Ovechkin's slap shot.
Early in the second, Holtby extended his right pad to make a similar stop on Jenner's rebound attempt following William Karlsson's slap shot.
NOTES: Washington's Andre Burakovsky crashed to the ground in a scary-looking second-period collision with teammate Oshie, but remained in the game. . Blue Jackets D Fedor Tyutin was scratched after playing all 10 of Columbus' previous games . Columbus C Ryan Johansen returned after missing two games due to illness. . Both teams continue back-to-back sets Saturday, with Washington traveling to Florida and Columbus hosting Winnipeg.