Extracting actionable insights from online dialogues is crucial for entities such as organizations, influencers, and digital platforms. However, analyzing online conversations, particularly for discerning stances (as in favor or oppose) from replies, remains challenging and demands sophisticated natural language processing (NLP) techniques. Recent advancements in Large Language Models (LLMs) present promising prospects for such tasks. However, their efficacy in analyzing online dialogues remains uncertain. This study introduces a unique dataset of Twitter posts and replies labeled for stances (favor/oppose/neutral) and proposes an approach to take advantage of LLMs for stance classification. When used with many simple machine learning classifiers, we find that text representations generated by LLMs outperform prior state-of-the-art approaches for stance classification, highlighting LLM's proficiency in generating high-quality text representations. Nevertheless, challenges arise when additional contextual information is essential to evaluate the stance. To tackle this, our approach integrates user-specific contextual information by generating network-based user embeddings, significantly improving performance. Our experiments reveal an 18\% enhancement in performance using LLM-generated text representations, further bolstered by an additional 5\% with contextual user information, as validated on a human-labeled dataset. This progress in stance classification within social media conversations underscores the potential synergy of LLMs and user-specific contextual data, offering practical insights for businesses and platforms.